Goroutines, Channels, WaitGroups, and select (just trying to understand) - go

Question

I am not necessarily trying to accomplish something specific, more just understand how goroutines, channels, waitgroups, and select (on channels) plays together. I am writing a simple program that loops through an slice of URLs, fetches the URL, then basically just ends. The simple idea is that I want all of the fetches to occur and return, send their data over channels, and then end once all fetches have occurred. I am almost there, and I know I am missing something in my select that will end the loop, something to say "hey the waitgroup is empty now", but I am unsure how to best do that. Mind taking a look and clearing it up for me? Right now everything runs just fine, it just doesn't terminate, so clearly I am missing something and/or not understanding how some of these components should work together.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"sync"
)
var urls = []string{
"https://www.google.com1",
"https://www.gentoo.org",
}
var wg sync.WaitGroup
// simple struct to store fetching
type urlObject struct {
url string
success bool
body string
}
func getPage(url string, channelMain chan urlObject, channelError chan error) {
// increment waitgroup, defer decrementing
wg.Add(1)
defer wg.Done()
fmt.Println("fetching " + url)
// create a urlObject
uO := urlObject{
url: url,
success: false,
}
// get URL
response, getError := http.Get(url)
// close response later on
if response != nil {
defer response.Body.Close()
}
// send error over error channel if one occurs
if getError != nil {
channelError <- getError
return
}
// convert body to []byte
body, conversionError := ioutil.ReadAll(response.Body)
// convert []byte to string
bodyString := string(body)
// if a conversion error happens send it over the error channel
if conversionError != nil {
channelError <- conversionError
} else {
// if not send a urlObject over the main channel
uO.success = true
uO.body = bodyString
channelMain <- uO
}
}
func main() {
var channelMain = make(chan urlObject)
var channelError = make(chan error)
for _, v := range urls {
go getPage(v, channelMain, channelError)
}
// wait on goroutines to finish
wg.Wait()
for {
select {
case uO := <-channelMain:
fmt.Println("completed " + uO.url)
case err := <-channelError:
fmt.Println("error: " + err.Error())
}
}
}

Answer 1

You need to make the following changes:
As people have mentioned, you probably want to call wg.Add(1) in the main function, before calling your goroutine. That way you KNOW it occurs before the defer wg.Done() call.
Your channel reads will block, unless you can figure out a way to either close the channels in your goroutines, or make them buffered. Probably the easiest way is to make them buffered, e.g., var channelMain = make(chan urlObject, len(urls))
The break in your select statement is going to only exit the select, not the containing for loop. You can label the for loop and break to that, or use some sort of conditional variable.
Playground link to working version: https://play.golang.org/p/WH1fm2MhP-L
package main
import (
"fmt"
"io/ioutil"
"net/http"
"sync"
)
var urls = []string{
"https://www.google.com1",
"https://www.gentoo.org",
}
var wg sync.WaitGroup
// simple struct to store fetching
type urlObject struct {
url string
success bool
body string
}
func getPage(url string, channelMain chan urlObject, channelError chan error) {
// increment waitgroup, defer decrementing
defer wg.Done()
fmt.Println("fetching " + url)
// create a urlObject
uO := urlObject{
url: url,
success: false,
}
// get URL
response, getError := http.Get(url)
// close response later on
if response != nil {
defer response.Body.Close()
}
// send error over error channel if one occurs
if getError != nil {
channelError <- getError
return
}
// convert body to []byte
body, conversionError := ioutil.ReadAll(response.Body)
// convert []byte to string
bodyString := string(body)
// if a conversion error happens send it over the error channel
if conversionError != nil {
channelError <- conversionError
} else {
// if not send a urlObject over the main channel
uO.success = true
uO.body = bodyString
channelMain <- uO
}
}
func main() {
var channelMain = make(chan urlObject, len(urls))
var channelError = make(chan error, len(urls))
for _, v := range urls {
wg.Add(1)
go getPage(v, channelMain, channelError)
}
// wait on goroutines to finish
wg.Wait()
for done := false; !done; {
select {
case uO := <-channelMain:
fmt.Println("completed " + uO.url)
case err := <-channelError:
fmt.Println("error: " + err.Error())
default:
done = true
}
}
}