I'm new to Golang, I would like to understand more about goroutine. I'll leave two examples and I wanted the opinion of which of the two is more performative and why?
func doRequest(method string, url string, body io.Reader) (*http.Response, error) {
request, _ := http.NewRequest(method, url, body)
response, err := c.httpClient.Do(request)
request.Close = true
c.httpClient.CloseIdleConnections()
return response, err
}
first:
func test() {
var wg *sync.WaitGroup = new(sync.WaitGroup)
qtd := 5
wg.Add(qtd)
for i := 0; i < qtd; i++ {
go func(wg *sync.WaitGroup) {
defer wg.Done()
doRequest(http.MethodGet, "http://test.com", nil)
}(wg)
}
wg.Wait()
}
Second:
func test() {
var wg *sync.WaitGroup = new(sync.WaitGroup)
wg.Add(1)
go func(wg *sync.WaitGroup) {
defer wg.Done()
for i := 0; i < 5; i++ {
doRequest(http.MethodGet, "http://test.com", nil)
}
}(wg)
wg.Wait()
}
Is there a better way than these two?
If not, which of the two is more performant?
The go keyword before a function will create a goroutine.
5 gouroutines, 1 doRequest() in per goroutine
1 gouroutine, 5 doRequest() in a goroutine
In the 1st case, 5 goroutines will run concurrently.
Check out concurrency in: A Tour of Go
Related
I am having issue while using waitgroup with the buffered channel. The problem is waitgroup closes before channel is read completely, which make my channel is half read and break in between.
func main() {
var wg sync.WaitGroup
var err error
start := time.Now()
students := make([]studentDetails, 0)
studentCh := make(chan studentDetail, 10000)
errorCh := make(chan error, 1)
wg.Add(1)
go s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
go func(ch chan studentDetail, e chan error) {
LOOP:
for {
select {
case p, ok := <-ch:
if ok {
L.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
} else {
L.Print("Closed channel")
break LOOP
}
case err = <-e:
if err != nil {
break
}
}
}
}(studentCh, errorCh)
wg.Wait()
close(studentCh)
close(errorCh)
L.Warnln("closed: all wait-groups completed!")
L.Warnf("total items fetched: %d", len(students))
elapsed := time.Since(start)
L.Warnf("operation took %s", elapsed)
}
The problem is this function is recursive. I mean some http call to fetch students and then make more calls depending on condition.
func (s Student) getDetailStudents(rCtx context.Context, content chan<- studentDetail, errorCh chan<- error, wg *sync.WaitGroup, url string, subSection bool) {
util.MustNotNil(rCtx)
L := logger.GetLogger(rCtx)
defer func() {
L.Println("Closing all waitgroup!")
wg.Done()
}()
wc := getWC()
httpClient := wc.Registry.MustHTTPClient()
res, err := httpClient.Get(url)
if err != nil {
L.Fatal(err)
}
defer res.Body.Close()
if res.StatusCode != 200 {
L.Errorf("status code error: %d %s", res.StatusCode, res.Status)
errorCh <- errors.New("service_status_code")
return
}
// parse response and return error if found some through errorCh as done above.
// decide page subSection based on response if it is more.
if !subSection {
wg.Add(1)
go s.getDetailStudents(rCtx, content, errorCh, wg, link, true)
// L.Warnf("total pages found %d", pageSub.Length()+1)
}
// Find students from response list and parse each Student
students := s.parseStudentItemList(rCtx, item)
for _, student := range students {
content <- student
}
L.Warnf("Calling HTTP Service for %q with total %d record", url, elementsSub.Length())
}
Variables are changed to avoid original code base.
The problem is students are read randomly as soon as Waitgroup complete. I am expecting to hold the execution until all students are read, In case of error it should break as soon error encounter.
You need to know when the receiving goroutine completes. The WaitGroup does that for the generating goroutine. So, you can use two waitgroups:
wg.Add(1)
go s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
wgReader.Add(1)
go func(ch chan studentDetail, e chan error) {
defer wgReader.Done()
...
}
wg.Wait()
close(studentCh)
close(errorCh)
wgReader.Wait() // Wait for the readers to complete
Since you are using buffered channels you can retrieve the remaining values after closing the channel. You will also need a mechanism to prevent your main function from exiting too early while the reader is still doing work ,as #Burak Serdar has advised.
I restructured the code to give a working example but it should get the point across.
package main
import (
"context"
"log"
"sync"
"time"
)
type studentDetails struct {
title string
link string
}
func main() {
var wg sync.WaitGroup
var err error
students := make([]studentDetails, 0)
studentCh := make(chan studentDetails, 10000)
errorCh := make(chan error, 1)
start := time.Now()
wg.Add(1)
go getDetailStudents(context.TODO(), studentCh, errorCh, &wg, "http://example.com", false)
LOOP:
for {
select {
case p, ok := <-studentCh:
if ok {
log.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
} else {
log.Println("Draining student channel")
for p := range studentCh {
log.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
}
break LOOP
}
case err = <-errorCh:
if err != nil {
break LOOP
}
case <-wrapWait(&wg):
close(studentCh)
}
}
close(errorCh)
elapsed := time.Since(start)
log.Printf("operation took %s", elapsed)
}
func getDetailStudents(rCtx context.Context, content chan<- studentDetails, errorCh chan<- error, wg *sync.WaitGroup, url string, subSection bool) {
defer func() {
log.Println("Closing")
wg.Done()
}()
if !subSection {
wg.Add(1)
go getDetailStudents(rCtx, content, errorCh, wg, url, true)
// L.Warnf("total pages found %d", pageSub.Length()+1)
}
content <- studentDetails{
title: "title",
link: "link",
}
}
// helper function to allow using WaitGroup in a select
func wrapWait(wg *sync.WaitGroup) <-chan struct{} {
out := make(chan struct{})
go func() {
wg.Wait()
out <- struct{}{}
}()
return out
}
wg.Add(1)
go func(){
defer wg.Done()
// I do not think that you need a recursive function.
// this function overcomplicated.
s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
}(...)
wg.Add(1)
go func(ch chan studentDetail, e chan error) {
defer wg.Done()
...
}(...)
wg.Wait()
close(studentCh)
close(errorCh)
This should solve the problem. s.getDetailStudents function must be simplified. Making it recursive does not have any benefit.
I want to write the contents of the process function with a function different from the main function.
I want to count the number of urls that did not result in an error with "resp, err: = client.Do (req)".
I want to write the number of successes before fmt.Println ("Finish!") In the main function.
What should I do?
func main() {
site_list := [][]string{
{"site1","https://www.aaaa"},
{"site2","https://www.bbbb"},
{"site3","https://www.cccc"},
{"site4","https://www.dddd"},
}
var wg sync.WaitGroup
maxChan := make(chan bool, 2)
for _, v := range site_list {
maxChan <- true
wg.Add(1)
go process(v[1], maxChan, &wg)
}
wg.Wait()
fmt.Println("Finish!")
}
func process(site_url string, maxChan chan bool, wg *sync.WaitGroup) {
defer wg.Done()
defer func(maxChan chan bool) {
<-maxChan
}(maxChan)
req, err := http.NewRequest("GET", site_url, nil)
client := new(http.Client)
resp, err := client.Do(req)
if err != nil {
fmt.Println("client.Doエラー" + site_url)
return
}
defer resp.Body.Close()
}
As per my comment above, create a custom work item struct:
type urlItem struct {
Name string
Url string
Err error
}
Use pointers, so that the process() function can update the underlying struct:
site_list := []*urlItem{
&urlItem{"site1", "https://www.aaaa", nil},
&urlItem{"site2", "https://www.bbbb", nil},
&urlItem{"site3", "https://www.cccc", nil},
&urlItem{"site4", "https://www.dddd", nil},
}
process() can then safely update it's particular work item (without the need for any mutexs or synchronization) as each go-routine gets a unique work item.
func process(site *urlItem, maxChan chan bool, wg *sync.WaitGroup) { /* ... */ }
Then tally the go-routine errors at the end:
var failCount int
for _, v := range site_list {
if v.Err != nil {
failCount++
}
}
Playground Example
May be I can not see obvious thing, what am I doing wrong:
func printSize (listOfUrls []string){
var wg sync.WaitGroup
wg.Add(len(listOfUrl))
for _, myurl := range(listOfUrls){
go func(){
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}()
}
wg.Wait()
}
If I remove wg and go, I receive the size of each url body correctly. If I do it as in above, it prints zeroes almost instantly. The getUrlBody() takes time to execute sometimes minutes.
Answering comment: I also tried it this way, to be sure, and it demonstrate same behaviour.
I found the error was in getUrlBody and main() function...
func printSize(listOfUrls []string) {
var wg sync.WaitGroup
wg.Add(len(listOfUrls))
for _, myurl := range listOfUrls {
go f(myurl, &wg)
}
wg.Wait()
}
func f(myurl string, wg *sync.WaitGroup) {
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}
All of the goroutines are sharing the single myurl variable. See https://golang.org/doc/faq#closures_and_goroutines for more information.
Change the code to:
func f(listOfUrls []string){
var wg sync.WaitGroup
wg.Add(len(listOfUrl))
for _, myurl := range(listOfUrls){
go func(myurl string){
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}(myurl)
}
wg.Wait()
}
or
func f(listOfUrls []string){
var wg sync.WaitGroup
wg.Add(len(listOfUrl))
for _, myurl := range(listOfUrls){
myurl := myurl
go func(){
body := getUrlBody(myurl)
fmt.Println(len(body))
wg.Done()
}()
}
wg.Wait()
}
I am newbie in Golang - While trying to rewrite the following to a single thread implementation.
.....
run := func(handler func(chan<- modify), threads int) <-chan modify {
result := make(chan modify)
go func() {
var wg sync.WaitGroup
for i := 0; i < threads; i++ {
wg.Add(1)
go func() {
defer wg.Done()
handler(result)
}()
}
wg.Wait()
close(result)
}()
return result
}
modifyAgg := run(func(result chan<- modify) {
aggre := run(func(result chan<- modify) {
u.addAgg(slices, result) // returns result channel
}, u.threads.GrpTxns)
....
In the above code the variable addAgg is of type chan<- modify. The following is not - get error while ranging over the variable aggre "cannot range over addAgg(type func())"
aggre := func() {
result:= make(chan modify)
defer close(result)
u.addAgg(slices, result) // returns result channel
}
How to change the second implementation to mimic the first one? Thank you !
I was able to implement this in single thread...
aggre := func() <-chan modify{
result:= make(chan modify, 50) // make it non blocking
u.addAgg(slices, result)
defer close(result)
return result
}()
I'm trying to write to a channel as last action in a goroutine function.
Unfortunately this is not working. And the waitGroup is never done.
import (
"sync"
"github.com/SlyMarbo/rss"
"fmt"
)
func main() {
urls := []string{"http://rss.cnn.com/rss/edition.rss", "http://rss.time.com/web/time/rss/top/index.xml"}
var c = make(chan string)
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go receiveRss(url, &wg, c)
}
wg.Wait()
fmt.Println("==============DONE=================")
}
func receiveRss(url string, wg *sync.WaitGroup, c chan string) {
defer wg.Done()
feed, err := rss.Fetch(url)
if err != nil {
fmt.Println("Failed to retrieve RSS feed", err)
}
items := feed.Items
for _, item := range items {
c <- item.Title
}
}
When replacing c <- item.Title with fmt.Println(item.Title) the deferred function is called and DONE is printed.
The problem is that I was only writing to the channel. Never reading from it.
Without doing this the channel would be useless.
The solution to this is:
Reading from the channel after the loop which starts the goroutines:
for title := range c {
fmt.Println(title)
}
This then causes an endless loop if the channel is never closed.
So I just close the channel after writing to it:
close(c)
Here is the whole code:
func main() {
urls := []string{"http://rss.cnn.com/rss/edition.rss", "http://rss.time.com/web/time/rss/top/index.xml"}
var c = make(chan []string)
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go receiveRss(url, &wg, c)
}
for title := range c {
fmt.Println(title)
}
wg.Wait()
fmt.Println("==============DONE=================")
}
func receiveRss(url string, wg *sync.WaitGroup, c chan []string) {
defer wg.Done()
feed, err := rss.Fetch(url)
if err != nil {
fmt.Println("Failed to retrieve RSS feed", err)
}
items := feed.Items
var titles []string
for _, item := range items {
titles = append(titles, item.Title)
}
c <- titles
close(c)
}