I'm trying to follow along the bounded goroutine example that is posted at http://blog.golang.org/pipelines/bounded.go. The problem that I'm having is that if there are more workers spun up then the amount of work to do, the extra workers never get cancelled. Everything else seems to work, the values get computed and logged, but when I close the groups channel, the workers just hang at the range statement.
I guess what I don't understand (in both my code and the example code) is how do the workers know when there is no more work to do and that they should exit?
Update
A working (i.e. non-working) example is posted at http://play.golang.org/p/T7zBCYLECp. It shows the deadlock on the workers since they are all asleep and there is no work to do. What I'm confused about is that I think the example code would have the same problem.
Here is the code that I'm currently using:
// Creates a pool of workers to do a bunch of computations
func computeAll() error {
done := make(chan struct{})
defer close(done)
groups, errc := findGroups(done)
// start a fixed number of goroutines to schedule with
const numComputers = 20
c := make(chan result)
var wg sync.WaitGroup
wg.Add(numComputers)
for i := 0; i < numComputers; i++ {
go func() {
compute(done, groups, c)
wg.Done()
}()
}
go func() {
wg.Wait()
close(c)
}()
// log the results of the computation
for r := range c { // log the results }
if err := <-errc; err != nil {
return err
}
return nil
}
Here is the code that fills up the channel with data:
// Retrieves the groups of data the must be computed
func findGroups(done <-chan struct{}) (<-chan model, <-chan error) {
groups := make(chan model)
errc := make(chan error, 1)
go func() {
// close the groups channel after find returns
defer close(groups)
group, err := //... code to get the group ...
if err == nil {
// add the group to the channel
select {
case groups <- group:
}
}
}()
return groups, errc
}
And here is the code that reads the channel to do the computations.
// Computes the results for the groups of data
func compute(done <-chan struct{}, groups <-chan model, c chan<- result) {
for group := range groups {
value := compute(group)
select {
case c <- result{value}:
case <-done:
return
}
}
}
Because you're trying to read from errc and it's empty unless there's an error.
//edit
computeAll() will always block on <- errc if there are no errors, another approach is to use something like:
func computeAll() (err error) {
.........
select {
case err = <-errc:
default: //don't block
}
return
}
Try to close the errc as OneOfOne says
go func() {
wg.Wait()
close(c)
close(errc)
}()
// log the results of the computation
for r := range c { // log the results }
if err := range errc {
if err != nil {
return err
}
}
Related
I am having issue while using waitgroup with the buffered channel. The problem is waitgroup closes before channel is read completely, which make my channel is half read and break in between.
func main() {
var wg sync.WaitGroup
var err error
start := time.Now()
students := make([]studentDetails, 0)
studentCh := make(chan studentDetail, 10000)
errorCh := make(chan error, 1)
wg.Add(1)
go s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
go func(ch chan studentDetail, e chan error) {
LOOP:
for {
select {
case p, ok := <-ch:
if ok {
L.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
} else {
L.Print("Closed channel")
break LOOP
}
case err = <-e:
if err != nil {
break
}
}
}
}(studentCh, errorCh)
wg.Wait()
close(studentCh)
close(errorCh)
L.Warnln("closed: all wait-groups completed!")
L.Warnf("total items fetched: %d", len(students))
elapsed := time.Since(start)
L.Warnf("operation took %s", elapsed)
}
The problem is this function is recursive. I mean some http call to fetch students and then make more calls depending on condition.
func (s Student) getDetailStudents(rCtx context.Context, content chan<- studentDetail, errorCh chan<- error, wg *sync.WaitGroup, url string, subSection bool) {
util.MustNotNil(rCtx)
L := logger.GetLogger(rCtx)
defer func() {
L.Println("Closing all waitgroup!")
wg.Done()
}()
wc := getWC()
httpClient := wc.Registry.MustHTTPClient()
res, err := httpClient.Get(url)
if err != nil {
L.Fatal(err)
}
defer res.Body.Close()
if res.StatusCode != 200 {
L.Errorf("status code error: %d %s", res.StatusCode, res.Status)
errorCh <- errors.New("service_status_code")
return
}
// parse response and return error if found some through errorCh as done above.
// decide page subSection based on response if it is more.
if !subSection {
wg.Add(1)
go s.getDetailStudents(rCtx, content, errorCh, wg, link, true)
// L.Warnf("total pages found %d", pageSub.Length()+1)
}
// Find students from response list and parse each Student
students := s.parseStudentItemList(rCtx, item)
for _, student := range students {
content <- student
}
L.Warnf("Calling HTTP Service for %q with total %d record", url, elementsSub.Length())
}
Variables are changed to avoid original code base.
The problem is students are read randomly as soon as Waitgroup complete. I am expecting to hold the execution until all students are read, In case of error it should break as soon error encounter.
You need to know when the receiving goroutine completes. The WaitGroup does that for the generating goroutine. So, you can use two waitgroups:
wg.Add(1)
go s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
wgReader.Add(1)
go func(ch chan studentDetail, e chan error) {
defer wgReader.Done()
...
}
wg.Wait()
close(studentCh)
close(errorCh)
wgReader.Wait() // Wait for the readers to complete
Since you are using buffered channels you can retrieve the remaining values after closing the channel. You will also need a mechanism to prevent your main function from exiting too early while the reader is still doing work ,as #Burak Serdar has advised.
I restructured the code to give a working example but it should get the point across.
package main
import (
"context"
"log"
"sync"
"time"
)
type studentDetails struct {
title string
link string
}
func main() {
var wg sync.WaitGroup
var err error
students := make([]studentDetails, 0)
studentCh := make(chan studentDetails, 10000)
errorCh := make(chan error, 1)
start := time.Now()
wg.Add(1)
go getDetailStudents(context.TODO(), studentCh, errorCh, &wg, "http://example.com", false)
LOOP:
for {
select {
case p, ok := <-studentCh:
if ok {
log.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
} else {
log.Println("Draining student channel")
for p := range studentCh {
log.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
}
break LOOP
}
case err = <-errorCh:
if err != nil {
break LOOP
}
case <-wrapWait(&wg):
close(studentCh)
}
}
close(errorCh)
elapsed := time.Since(start)
log.Printf("operation took %s", elapsed)
}
func getDetailStudents(rCtx context.Context, content chan<- studentDetails, errorCh chan<- error, wg *sync.WaitGroup, url string, subSection bool) {
defer func() {
log.Println("Closing")
wg.Done()
}()
if !subSection {
wg.Add(1)
go getDetailStudents(rCtx, content, errorCh, wg, url, true)
// L.Warnf("total pages found %d", pageSub.Length()+1)
}
content <- studentDetails{
title: "title",
link: "link",
}
}
// helper function to allow using WaitGroup in a select
func wrapWait(wg *sync.WaitGroup) <-chan struct{} {
out := make(chan struct{})
go func() {
wg.Wait()
out <- struct{}{}
}()
return out
}
wg.Add(1)
go func(){
defer wg.Done()
// I do not think that you need a recursive function.
// this function overcomplicated.
s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
}(...)
wg.Add(1)
go func(ch chan studentDetail, e chan error) {
defer wg.Done()
...
}(...)
wg.Wait()
close(studentCh)
close(errorCh)
This should solve the problem. s.getDetailStudents function must be simplified. Making it recursive does not have any benefit.
I have two goroutines: the main worker and a helper that it spins off for some help. helper can encounter errors, so I use a channel to communicate errors over from the helper to the worker.
func helper(c chan <- error) (){
//do some work
c <- err // send errors/nil on c
}
Here is how helper() is called:
func worker() error {
//do some work
c := make(chan error, 1)
go helper(c)
err := <- c
return err
}
Questions:
Is the statement err := <- c blocking worker? I don't think so, since the channel is buffered.
If it is blocking, how do I make it non-blocking? My requirement is to have worker and its caller continue with rest of the work, without waiting for the value to appear on the channel.
Thanks.
You can easily verify
func helper(c chan<- error) {
time.Sleep(5 * time.Second)
c <- errors.New("") // send errors/nil on c
}
func worker() error {
fmt.Println("do one")
c := make(chan error, 1)
go helper(c)
err := <-c
fmt.Println("do two")
return err
}
func main() {
worker()
}
Q: Is the statement err := <- c blocking worker? I don't think so, since the channel is buffered.
A: err := <- c will block worker.
Q: If it is blocking, how do I make it non-blocking? My requirement is to have worker and its caller continue with rest of the work, without waiting for the value to appear on the channel.
A: If you don't want blocking, just remove err := <-c. If you need err at the end, just move err := <-c to the end.
You can not read channel without blocking, if you go through without blocking, can can no more exec this code, unless your code is in a loop.
Loop:
for {
select {
case <-c:
break Loop
default:
//default will go through without blocking
}
// do something
}
And have you ever seen errgroup or waitgroup？
It use atomic, cancel context and sync.Once to implement this.
https://github.com/golang/sync/blob/master/errgroup/errgroup.go
https://github.com/golang/go/blob/master/src/sync/waitgroup.go
Or you can just use it, go you func and then wait for error in any place you want.
In your code, the rest of the work is independent of whether the helper encountered an error. You can simply receive from the channel after the rest of the work is completed.
func worker() error {
//do some work
c := make(chan error, 1)
go helper(c)
//do rest of the work
return <-c
}
I think you need this code..
run this code
package main
import (
"log"
"sync"
)
func helper(c chan<- error) {
for {
var err error = nil
// do job
if err != nil {
c <- err // send errors/nil on c
break
}
}
}
func worker(c chan error) error {
log.Println("first log")
go func() {
helper(c)
}()
count := 1
Loop:
for {
select {
case err := <- c :
return err
default:
log.Println(count, " log")
count++
isFinished := false
// do your job
if isFinished {
break Loop // remove this when you test
}
}
}
return nil
}
func main() {
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
c := make(chan error, 1)
worker(c)
wg.Done()
}()
wg.Wait()
}
In go I have two callbacks that eventually do not fire.
registerCb(func() {...})
registerCb(func() {...})
/* Wait for both func to execute with timeout */
I want to wait for both of them but having a timeout if one is not executed.
sync.WaitGroup does not work, since it is blocking and not channel based. Also you call WaitGroup.Done() without the risk of panic outside the callbacks.
My current solution is using just two booleans and a busy wait loop. But that's not satisfying.
Is there any idiomatic way that do not use polling or busy waiting?
Update:
Here is some code that demonstrates a busy wait solution but should return as soon as both callbacks are fired or after the timeout, without using polling
package main
import (
"fmt"
"log"
"sync"
"time"
)
var cbOne func()
var cbTwo func()
func registerCbOne(cb func()) {
cbOne = cb
}
func registerCbTwo(cb func()) {
cbTwo = cb
}
func executeCallbacks() {
<-time.After(1 * time.Second)
cbOne()
// Might never happen
//<-time.After(1 * time.Second)
//cbTwo()
}
func main() {
// Some process in background will execute our callbacks
go func() {
executeCallbacks()
}()
err := WaitAllOrTimeout(3 * time.Second)
if err != nil {
fmt.Println("Error: ", err.Error())
}
fmt.Println("Hello, playground")
}
func WaitAllOrTimeout(to time.Duration) error {
cbOneDoneCh := make(chan bool, 1)
cbTwoDoneCh := make(chan bool, 1)
cbOneDone := false
cbTwoDone := false
registerCbOne(func() {
fmt.Println("cb One");
cbOneDoneCh <- true
})
registerCbTwo(func() {
fmt.Println("cb Two");
cbTwoDoneCh <- true
})
// Wait for cbOne and cbTwo to be executed or a timeout
// Busywait solution
for {
select {
case <-time.After(to):
if cbOneDone && cbTwoDone {
fmt.Println("Both CB executed (we could poll more often)")
return nil
}
fmt.Println("Timeout!")
return fmt.Errorf("Timeout")
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
}
This is a followup to my comment, added after you added your example solution. To be clearer than I can in comments, your example code is actually not that bad. Here is your original example:
// Busywait solution
for {
select {
case <-time.After(to):
if cbOneDone && cbTwoDone {
fmt.Println("Both CB executed (we could poll more often)")
return nil
}
fmt.Println("Timeout!")
return fmt.Errorf("Timeout")
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
This isn't a "busy wait" but it does have several bugs (including the fact that you need an only-once send semantic for the done channels, or maybe easier and at least as good, to just close them once when done, perhaps using sync.Once). What we want to do is:
Start a timer with to as the timeout.
Enter a select loop, using the timer's channel and the two "done" channels.
We want to exit the select loop when the first of the following events occurs:
the timer fires, or
both "done" channels have been signaled.
If we're going to close the two done channels we'll want to have the Ch variables cleared (set to nil) as well so that the selects don't spin—that would turn this into a true busy-wait—but for the moment let's just assume instead that we send exactly once on them on callback, and otherwise just leak the channels, so that we can use your code as written as those selects will only ever return once. Here's the updated code:
t := timer.NewTimer(to)
for !cbOneDone || !cbTwoDone {
select {
case <-t.C:
fmt.Println("Timeout!")
return fmt.Errorf("timeout")
}
case <-cbOneDoneCh:
cbOneDone = true
case <-cbTwoDoneCh:
cbTwoDone = true
}
}
// insert t.Stop() and receive here to drain t.C if desired
fmt.Println("Both CB executed")
return nil
Note that we will go through the loop at most two times:
If we receive from both Done channels, once each, the loop stops without a timeout. There's no spinning/busy-waiting: we never received anything from t.C. We return nil (no error).
If we receive from one Done channel, the loop resumes but blocks waiting for the timer or the other Done channel.
If we ever receive from t.C, it means we didn't get both callbacks yet. We may have had one, but there's been a timeout and we choose to give up, which was our goal. We return an error, without going back through the loop.
A real version needs a bit more work to clean up properly and avoid leaking "done" channels (and the timer channel and its goroutine; see comment), but this is the general idea. You're already turning the callbacks into channel operations, and you already have a timer with its channel.
func wait(ctx context.Context, wg *sync.WaitGroup) error {
done := make(chan struct{}, 1)
go func() {
wg.Wait()
done <- struct{}{}
}()
select {
case <-done:
// Counter is 0, so all callbacks completed.
return nil
case <-ctx.Done():
// Context cancelled.
return ctx.Err()
}
}
Alternatively, you can pass a time.Duration and block on <-time.After(d) rather than on <-ctx.Done(), but I would argue that using context is more idiomatic.
below code present two variations,
the first is the regular pattern, nothing fancy, it does the job and does it well. You launch your callbacks into a routine, you make them push to a sink, listen that sink for a result or timeout. Take care to the sink channel initial capacity, to prevent leaking a routine it must match the number of callbacks.
the second factories out the synchronization mechanisms into small functions to assemble, two wait methods are provided, waitAll and waitOne. Nice to write, but definitely less efficient, more allocations, more back and forth with more channels, more complex to reason about, more subtle.
package main
import (
"fmt"
"log"
"sync"
"time"
)
func main() {
ExampleOne()
ExampleTwo()
ExampleThree()
fmt.Println("Hello, playground")
}
func ExampleOne() {
log.Println("start reg")
errs := make(chan error, 2)
go func() {
fn := callbackWithOpts("reg: so slow", 2*time.Second, nil)
errs <- fn()
}()
go func() {
fn := callbackWithOpts("reg: too fast", time.Millisecond, fmt.Errorf("broke!"))
errs <- fn()
}()
select {
case err := <-errs: // capture only one result,
// the fastest to finish.
if err != nil {
log.Println(err)
}
case <-time.After(time.Second): // or wait that many amount of time,
// in case they are all so slow.
}
log.Println("done reg")
}
func ExampleTwo() {
log.Println("start wait")
errs := waitAll(
withTimeout(time.Second,
callbackWithOpts("waitAll: so slow", 2*time.Second, nil),
),
withTimeout(time.Second,
callbackWithOpts("waitAll: too fast", time.Millisecond, nil),
),
)
for err := range trim(errs) {
if err != nil {
log.Println(err)
}
}
log.Println("done wait")
}
func ExampleThree() {
log.Println("start waitOne")
errs := waitOne(
withTimeout(time.Second,
callbackWithOpts("waitOne: so slow", 2*time.Second, nil),
),
withTimeout(time.Second,
callbackWithOpts("waitOne: too fast", time.Millisecond, nil),
),
)
for err := range trim(errs) {
if err != nil {
log.Println(err)
}
}
log.Println("done waitOne")
}
// a configurable callback for playing
func callbackWithOpts(msg string, tout time.Duration, err error) func() error {
return func() error {
<-time.After(tout)
fmt.Println(msg)
return err
}
}
// withTimeout return a function that returns first error or times out and return nil
func withTimeout(tout time.Duration, h func() error) func() error {
return func() error {
d := make(chan error, 1)
go func() {
d <- h()
}()
select {
case err := <-d:
return err
case <-time.After(tout):
}
return nil
}
}
// wait launches all func() and return their errors into the returned error channel; (merge)
// It is the caller responsability to drain the output error channel.
func waitAll(h ...func() error) chan error {
d := make(chan error, len(h))
var wg sync.WaitGroup
for i := 0; i < len(h); i++ {
wg.Add(1)
go func(h func() error) {
defer wg.Done()
d <- h()
}(h[i])
}
go func() {
wg.Wait()
close(d)
}()
return d
}
// wait launches all func() and return the first error into the returned error channel
// It is the caller responsability to drain the output error channel.
func waitOne(h ...func() error) chan error {
d := make(chan error, len(h))
one := make(chan error, 1)
var wg sync.WaitGroup
for i := 0; i < len(h); i++ {
wg.Add(1)
go func(h func() error) {
defer wg.Done()
d <- h()
}(h[i])
}
go func() {
for err := range d {
one <- err
close(one)
break
}
}()
go func() {
wg.Wait()
close(d)
}()
return one
}
func trim(err chan error) chan error {
out := make(chan error)
go func() {
for e := range err {
out <- e
}
close(out)
}()
return out
}
I try to build concurrent crawler based on Tour and some others SO answers regarding that. What I have currently is below but I think I have here two subtle issues.
Sometimes I get 16 urls in response and sometimes 17 (debug print in main). I know it because when I even change WriteToSlice to Read then in Read sometimes 'Read: end, counter = ' is never reached and it's always when I get 16 urls.
I have troubles with err channel, I get no messages in this channel, even when I run my main Crawl method with address like www.golang.org so without valid schema error should be send via err channel
Concurrency is really difficult topic, help and advice will be appreciated
package main
import (
"fmt"
"net/http"
"sync"
"golang.org/x/net/html"
)
type urlCache struct {
urls map[string]struct{}
sync.Mutex
}
func (v *urlCache) Set(url string) bool {
v.Lock()
defer v.Unlock()
_, exist := v.urls[url]
v.urls[url] = struct{}{}
return !exist
}
func newURLCache() *urlCache {
return &urlCache{
urls: make(map[string]struct{}),
}
}
type results struct {
data chan string
err chan error
}
func newResults() *results {
return &results{
data: make(chan string, 1),
err: make(chan error, 1),
}
}
func (r *results) close() {
close(r.data)
close(r.err)
}
func (r *results) WriteToSlice(s *[]string) {
for {
select {
case data := <-r.data:
*s = append(*s, data)
case err := <-r.err:
fmt.Println("e ", err)
}
}
}
func (r *results) Read() {
fmt.Println("Read: start")
counter := 0
for c := range r.data {
fmt.Println(c)
counter++
}
fmt.Println("Read: end, counter = ", counter)
}
func crawl(url string, depth int, wg *sync.WaitGroup, cache *urlCache, res *results) {
defer wg.Done()
if depth == 0 || !cache.Set(url) {
return
}
response, err := http.Get(url)
if err != nil {
res.err <- err
return
}
defer response.Body.Close()
node, err := html.Parse(response.Body)
if err != nil {
res.err <- err
return
}
urls := grablUrls(response, node)
res.data <- url
for _, url := range urls {
wg.Add(1)
go crawl(url, depth-1, wg, cache, res)
}
}
func grablUrls(resp *http.Response, node *html.Node) []string {
var f func(*html.Node) []string
var results []string
f = func(n *html.Node) []string {
if n.Type == html.ElementNode && n.Data == "a" {
for _, a := range n.Attr {
if a.Key != "href" {
continue
}
link, err := resp.Request.URL.Parse(a.Val)
if err != nil {
continue
}
results = append(results, link.String())
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
f(c)
}
return results
}
res := f(node)
return res
}
// Crawl ...
func Crawl(url string, depth int) []string {
wg := &sync.WaitGroup{}
output := &[]string{}
visited := newURLCache()
results := newResults()
defer results.close()
wg.Add(1)
go crawl(url, depth, wg, visited, results)
go results.WriteToSlice(output)
// go results.Read()
wg.Wait()
return *output
}
func main() {
r := Crawl("https://www.golang.org", 2)
// r := Crawl("www.golang.org", 2) // no schema, error should be generated and send via err
fmt.Println(len(r))
}
Both your questions 1 and 2 are a result of the same bug.
In Crawl() you are not waiting for this go routine to finish: go results.WriteToSlice(output). On the last crawl() function, the wait group is released, the output is returned and printed before the WriteToSlice function finishes with the data and err channel. So what has happened is this:
crawl() finishes, placing data in results.data and results.err.
Waitgroup wait() unblocks, causing main() to print the length of the result []string
WriteToSlice adds the last data (or err) item to the channel
You need to return from Crawl() not only when the data is done being written to the channel, but also when the channel is done being read in it's entirety (including the buffer). A good way to do this is close channels when you are sure that you are done with them. By organizing your code this way, you can block on the go routine that is draining the channels, and instead of using the wait group to release to main, you wait until the channels are 100% done.
You can see this gobyexample https://gobyexample.com/closing-channels. Remember that when you close a channel, the channel can still be used until the last item is taken. So you can close a buffered channel, and the reader will still get all the items that were queued in the channel.
There is some code structure that can change to make this cleaner, but here is a quick way to fix your program. Change Crawl to block on WriteToSlice. Close the data channel when the crawl function finishes, and wait for WriteToSlice to finish.
// Crawl ...
func Crawl(url string, depth int) []string {
wg := &sync.WaitGroup{}
output := &[]string{}
visited := newURLCache()
results := newResults()
go func() {
wg.Add(1)
go crawl(url, depth, wg, visited, results)
wg.Wait()
// All data is written, this makes `WriteToSlice()` unblock
close(results.data)
}()
// This will block until results.data is closed
results.WriteToSlice(output)
close(results.err)
return *output
}
Then on write to slice, you have to check for the closed channel to exit the for loop:
func (r *results) WriteToSlice(s *[]string) {
for {
select {
case data, open := <-r.data:
if !open {
return // All data done
}
*s = append(*s, data)
case err := <-r.err:
fmt.Println("e ", err)
}
}
}
Here is the full code: https://play.golang.org/p/GBpGk-lzrhd (it won't work in the playground)
I have the following function which spins off a given amount of go routines
func (r *Runner) Execute() {
var wg sync.WaitGroup
wg.Add(len(r.pipelines))
for _, p := range r.pipelines {
go executePipeline(p, &wg)
}
wg.Wait()
errs := ....//contains list of errors reported by any/all go routines
}
I was thinking there might be some way with channels, but I can't seem to figure it out.
One way to do this is using mutexes if you can make executePipeline retuen errors:
// ...
for _, p := range r.pipelines {
go func(p pipelineType) {
if err := executePipeline(p, &wg); err != nil {
mu.Lock()
errs = append(errs, err)
mu.UnLock()
}
}(p)
}
To use a channel, you can have a separate goroutine listning for errors:
errCh := make(chan error)
go func() {
for e := range errCh {
errs = append(errs, e)
}
}
and in the Execute function, make the following changes:
// ...
wg.Add(len(r.pipelines))
for _, p := range r.pipelines {
go func(p pipelineType) {
if err := executePipeline(p, &wg); err != nil {
errCh <- err
}
}(p)
}
wg.Wait()
close(errCh)
You can always use #zerkms method listed above if the number of goroutines is not high.
instead of returning error from executePipleline and using a anonymous function wrapper, you can always make above changes within the function itself.
You can use channels as #Kaveh Shahbazian suggested:
func (r *Runner) Execute() {
pipelineChan := makePipeline(r.pipelines)
for cnt := 0; cnt < len(r.pipelines); cnt++{
//recieve from channel
p := <- pipelineChan
//do something with the result
}
}
func makePipeline(pipelines []pipelineType) <-chan pipelineType{
pipelineChan := make(chan pipelineType)
go func(){
for _, p := range pipelines {
go func(p pipelineType){
pipelineChan <- executePipeline(p)
}(p)
}
}()
return pipelineChan
}
Please see this example: https://gist.github.com/steven-ferrer/9b2eeac3eed3f7667e8976f399d0b8ad