I am having issue while using waitgroup with the buffered channel. The problem is waitgroup closes before channel is read completely, which make my channel is half read and break in between.
func main() {
var wg sync.WaitGroup
var err error
start := time.Now()
students := make([]studentDetails, 0)
studentCh := make(chan studentDetail, 10000)
errorCh := make(chan error, 1)
wg.Add(1)
go s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
go func(ch chan studentDetail, e chan error) {
LOOP:
for {
select {
case p, ok := <-ch:
if ok {
L.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
} else {
L.Print("Closed channel")
break LOOP
}
case err = <-e:
if err != nil {
break
}
}
}
}(studentCh, errorCh)
wg.Wait()
close(studentCh)
close(errorCh)
L.Warnln("closed: all wait-groups completed!")
L.Warnf("total items fetched: %d", len(students))
elapsed := time.Since(start)
L.Warnf("operation took %s", elapsed)
}
The problem is this function is recursive. I mean some http call to fetch students and then make more calls depending on condition.
func (s Student) getDetailStudents(rCtx context.Context, content chan<- studentDetail, errorCh chan<- error, wg *sync.WaitGroup, url string, subSection bool) {
util.MustNotNil(rCtx)
L := logger.GetLogger(rCtx)
defer func() {
L.Println("Closing all waitgroup!")
wg.Done()
}()
wc := getWC()
httpClient := wc.Registry.MustHTTPClient()
res, err := httpClient.Get(url)
if err != nil {
L.Fatal(err)
}
defer res.Body.Close()
if res.StatusCode != 200 {
L.Errorf("status code error: %d %s", res.StatusCode, res.Status)
errorCh <- errors.New("service_status_code")
return
}
// parse response and return error if found some through errorCh as done above.
// decide page subSection based on response if it is more.
if !subSection {
wg.Add(1)
go s.getDetailStudents(rCtx, content, errorCh, wg, link, true)
// L.Warnf("total pages found %d", pageSub.Length()+1)
}
// Find students from response list and parse each Student
students := s.parseStudentItemList(rCtx, item)
for _, student := range students {
content <- student
}
L.Warnf("Calling HTTP Service for %q with total %d record", url, elementsSub.Length())
}
Variables are changed to avoid original code base.
The problem is students are read randomly as soon as Waitgroup complete. I am expecting to hold the execution until all students are read, In case of error it should break as soon error encounter.
You need to know when the receiving goroutine completes. The WaitGroup does that for the generating goroutine. So, you can use two waitgroups:
wg.Add(1)
go s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
wgReader.Add(1)
go func(ch chan studentDetail, e chan error) {
defer wgReader.Done()
...
}
wg.Wait()
close(studentCh)
close(errorCh)
wgReader.Wait() // Wait for the readers to complete
Since you are using buffered channels you can retrieve the remaining values after closing the channel. You will also need a mechanism to prevent your main function from exiting too early while the reader is still doing work ,as #Burak Serdar has advised.
I restructured the code to give a working example but it should get the point across.
package main
import (
"context"
"log"
"sync"
"time"
)
type studentDetails struct {
title string
link string
}
func main() {
var wg sync.WaitGroup
var err error
students := make([]studentDetails, 0)
studentCh := make(chan studentDetails, 10000)
errorCh := make(chan error, 1)
start := time.Now()
wg.Add(1)
go getDetailStudents(context.TODO(), studentCh, errorCh, &wg, "http://example.com", false)
LOOP:
for {
select {
case p, ok := <-studentCh:
if ok {
log.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
} else {
log.Println("Draining student channel")
for p := range studentCh {
log.Printf("Links %s: [%s]\n", p.title, p.link)
students = append(students, p)
}
break LOOP
}
case err = <-errorCh:
if err != nil {
break LOOP
}
case <-wrapWait(&wg):
close(studentCh)
}
}
close(errorCh)
elapsed := time.Since(start)
log.Printf("operation took %s", elapsed)
}
func getDetailStudents(rCtx context.Context, content chan<- studentDetails, errorCh chan<- error, wg *sync.WaitGroup, url string, subSection bool) {
defer func() {
log.Println("Closing")
wg.Done()
}()
if !subSection {
wg.Add(1)
go getDetailStudents(rCtx, content, errorCh, wg, url, true)
// L.Warnf("total pages found %d", pageSub.Length()+1)
}
content <- studentDetails{
title: "title",
link: "link",
}
}
// helper function to allow using WaitGroup in a select
func wrapWait(wg *sync.WaitGroup) <-chan struct{} {
out := make(chan struct{})
go func() {
wg.Wait()
out <- struct{}{}
}()
return out
}
wg.Add(1)
go func(){
defer wg.Done()
// I do not think that you need a recursive function.
// this function overcomplicated.
s.getDetailStudents(rCtx, studentCh , errorCh, &wg, s.Link, false)
}(...)
wg.Add(1)
go func(ch chan studentDetail, e chan error) {
defer wg.Done()
...
}(...)
wg.Wait()
close(studentCh)
close(errorCh)
This should solve the problem. s.getDetailStudents function must be simplified. Making it recursive does not have any benefit.
In below code snipped I'm trying to loop some data and for each iteration start new go routine. I have created error channel and if some errors appear goroutine it should write it to the channel. So I have 2 questions:
How to catch if all goroutines done and close error channel for breaking from "for" loop to continue main function execution?
if in each routine I'll call some functions which appends values to global variables will there be any issues with race condition? If yes, how to handle this.
package main
import (
"fmt"
)
var errc = make(chan error, 15)
func fetchAll() {
var N = 15
for i := 0; i < N; i++ {
go func(i int) {
err := error(nil)
if err != nil {
errc <- err
}
// Doing some stuff there
fmt.Println(i)
}(i)
}
}
func main() {
fetchAll()
for {
select {
case err := <-errc:
fmt.Println(err)
break
}
}
fmt.Println("Finished All")
}
Use a WaitGroup:
wg:=sync.WaitGroup{}
for i := 0; i < N; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
err := error(nil)
if err != nil {
errc <- err
}
// Doing some stuff there
fmt.Println(i)
}(i)
}
go func() {
wg.Wait()
close(errc)
}
Then you can change the loop in main. This loop will terminate when the channel is closed.
for err:=range errc {
fmt.Println(err)
}
If you have variables shared between goroutines, use a sync.Mutex shared among goroutines to control read/write to those variables, otherwise you'll run into race conditions.
I have the following program which serves as a proof of concept. I'm attempting to aggregate the results from chann, that is, too merge each instance of chann in to a common slice. Is this possible with my approach?
So my output for the following example would be a slice containing the following (in any order): []int{0,1,2}, thanks.
func DoStuff(i int, chann chan[]int, wg *sync.WaitGroup) {
defer wg.Done()
chann <-[]int{i}
}
func main() {
var wg sync.WaitGroup
chann := make(chan int[], 3)
defer close(chann)
for i := 0; i < count; 3 {
wg.Add(1)
go DoStuff(i, chann, &wg)
}
wg.Wait()
for {
select {
case result := <-chann:
fmt.Println(result)
os.Exit(1)
}
}
return nil
}
what you want to do is possible, but your program will not run because you are reading from the channel after wg.Wait(), so all goroutines will stop waiting to write, because you will never read from the channel.
You can read from the channel in a goroutine:
for i := 0; i < count; 3 {
wg.Add(1)
go DoStuff(i, chann, &wg)
}
}
go func() {
for data:=range chann {
// Process data
}
}()
wg.Wait()
// Close here, so the reading goroutine can terminate
close(chann)
I am newbie in Golang - While trying to rewrite the following to a single thread implementation.
.....
run := func(handler func(chan<- modify), threads int) <-chan modify {
result := make(chan modify)
go func() {
var wg sync.WaitGroup
for i := 0; i < threads; i++ {
wg.Add(1)
go func() {
defer wg.Done()
handler(result)
}()
}
wg.Wait()
close(result)
}()
return result
}
modifyAgg := run(func(result chan<- modify) {
aggre := run(func(result chan<- modify) {
u.addAgg(slices, result) // returns result channel
}, u.threads.GrpTxns)
....
In the above code the variable addAgg is of type chan<- modify. The following is not - get error while ranging over the variable aggre "cannot range over addAgg(type func())"
aggre := func() {
result:= make(chan modify)
defer close(result)
u.addAgg(slices, result) // returns result channel
}
How to change the second implementation to mimic the first one? Thank you !
I was able to implement this in single thread...
aggre := func() <-chan modify{
result:= make(chan modify, 50) // make it non blocking
u.addAgg(slices, result)
defer close(result)
return result
}()
How to handle crashes in a waitgroup gracefully?
In other words, in the following snippet of code, how to catch the panics/crashes of goroutines invoking method do()?
func do(){
str := "abc"
fmt.Print(str[3])
defer func() {
if err := recover(); err != nil {
fmt.Print(err)
}
}()
}
func main() {
var wg sync.WaitGroup
for i := 0; i < 1; i++ {
wg.Add(1)
go do()
defer func() {
wg.Done()
if err := recover(); err != nil {
fmt.Print(err)
}
}()
}
wg.Wait()
fmt.Println("This line should be printed after all those invocations fail.")
}
First, registering a deferred function to recover should be the first line in the function, as since you do it last, it won't even be reached because the line / code before the defer already panics and so the deferred function does not get registered which would restore the panicing state.
So change your do() function to this:
func do() {
defer func() {
if err := recover(); err != nil {
fmt.Println("Restored:", err)
}
}()
str := "abc"
fmt.Print(str[3])
}
Second: this alone will not make your code work, as you call wg.Defer() in a deferred function which would only run once main() finishes - which is never because you call wg.Wait() in your main(). So wg.Wait() waits for the wg.Done() calls, but wg.Done() calls will not be run until wg.Wait() returnes. It's a deadlock.
You should call wg.Done() from the do() function, in the deferred function, something like this:
var wg sync.WaitGroup
func do() {
defer func() {
if err := recover(); err != nil {
fmt.Println(err)
}
wg.Done()
}()
str := "abc"
fmt.Print(str[3])
}
func main() {
for i := 0; i < 1; i++ {
wg.Add(1)
go do()
}
wg.Wait()
fmt.Println("This line should be printed after all those invocations fail.")
}
Output (try it on the Go Playground):
Restored: runtime error: index out of range
This line should be printed after all those invocations fail.
This of course needed to move the wg variable to global scope. Another option would be to pass it to do() as an argument. If you decide to go this way, note that you have to pass a pointer to WaitGroup, else only a copy will be passed (WaitGroup is a struct type) and calling WaitGroup.Done() on a copy will not have effect on the original.
With passing WaitGroup to do():
func do(wg *sync.WaitGroup) {
defer func() {
if err := recover(); err != nil {
fmt.Println("Restored:", err)
}
wg.Done()
}()
str := "abc"
fmt.Print(str[3])
}
func main() {
var wg sync.WaitGroup
for i := 0; i < 1; i++ {
wg.Add(1)
go do(&wg)
}
wg.Wait()
fmt.Println("This line should be printed after all those invocations fail.")
}
Output is the same. Try this variant on the Go Playground.
#icza did a fantastic job explaining how to appropriately use WaitGroup and its functions Wait and Done
I like WaitGroup simplicity. However, I do not like that we need to pass the reference to the goroutine because that would mean that the concurrency logic would be mixed with your business logic.
So I came up with this generic function to solve this problem for me:
// Parallelize parallelizes the function calls
func Parallelize(functions ...func()) {
var waitGroup sync.WaitGroup
waitGroup.Add(len(functions))
defer waitGroup.Wait()
for _, function := range functions {
go func(copy func()) {
defer waitGroup.Done()
copy()
}(function)
}
}
So your example could be solved this way:
func do() {
defer func() {
if err := recover(); err != nil {
fmt.Println(err)
}
}()
str := "abc"
fmt.Print(str[3])
}
func main() {
Parallelize(do, do, do)
fmt.Println("This line should be printed after all those invocations fail.")
}
If you would like to use it, you can find it here https://github.com/shomali11/util