I have two go routine functions, one creates data and one adds the data to the database. When I'm done creating data I want to finish adding data but I don't know how many times I'm going to call the function.
for {
select {
case c <- dataChan:
go func() {
if data == false {
return // keeps the program running but never exits
}
createData(c.Data) // sends data to dataChan until data == false
}
go func() {
addData(c.Data) // need to keep the program running until all these goroutines finish
}
}
}
createData() creates and sends data to the dataChan and based on some variables it will eventually set data to false and cause the program to be held open by returning over and over again and stops creating more data, but I feel like there should be a cleaner way that allows me to exit when all the addData() goroutines finish. I have seen people use a channel for this but I don't know exactly how many times I'm going to call the function.
UPDATE: WORKING CODE
var wg sync.WaitGroup
for {
select {
case c <- dataChan:
wg.Add(1)
go func() {
if data == false {
wg.Wait()
os.Exit(0)
}
createData(c.Data)
}
go func() {
addData(c.Data)
defer wg.Done()
}
}
}
sync.WaitGroup is what you want here. You wg.Add(1) before starting each goroutine (or wg.Add(n) if you know the count up front), wg.Done() in each routine when finished, and wg.Wait() in main to wait untill all finish. The linked docs have an example, and as the docs note, copying it won't do what you want; you may want to make the variable a pointer (wg := new(sync.WaitGroup)).
Related
I have to update thousands of structs with datas available on some remote servers. So, I have to deal with thousands of Goroutines querying these remote servers (http requests or db requests) to update the struct with the responses. But the update (or not) of the struct is depending of the results of other structs.
So I imagined a simple code in which the goroutines are running, each of them performs its own request, put the result in a global struct that contain any information retrieved by the goroutines and informs the main func that the first part of the job is done (waiting for the signal that every goroutines do the same before deciding of updating or not its struct.)
The goroutine should now waits for a signal of the main thread that all goroutines are done before deciding updating or not
the simplified code looks like that :
type struct StructToUpdate {
//some properties
}
type struct GlobalWatcher {
//some userful informations for all structs to update (or not)
}
func main() {
//retrieving all structs to update
structsToUpdates := foo()
//launching goroutines
c := make(chan bool)
for _,structToUpdate := range structsToUpdates {
go updateStruct(&structToUpdate,c)
}
//waiting for all goroutines do their first part of job
for i:=0; i<len(structsToUpdates); i++ {
<- c
}
//HERE IS THE CODE TO INFORM GOROUTINES THEY CAN RESUME
}
func updateStruct(s *StructToUpdate, c chan bool) {
result := performSomeRequest(s)
informGlobalWatcherOfResult(result)
c <- true //I did my job
//HERE IS THE CODE TO WAIT FOR THE SIGNAL TO RESUME
}
The question is : What the more performant / idiomatic / elegant wait to :
send a signal from the main script ?
wait for this signal from the goroutine ?
I can imagine 3 ways to do this
in a first way, I can imagine a global bool var resume := false that will be turned to true by the main func when all goroutines do the first part of job. In this case, each goroutine can use an ugly for !resume { continue }....
in a more idiomatic code, I can imagine do the same thing but instead of using a bool, I can use a context.WithValue(ctx, "resume", false) and pass it to goroutine, but I still have a for !ctx.Value("resume")
in a last more elegant code, I can imagine using another resume := make(chan bool) passed to the goroutine. The main func could inform goroutines to resume closing this chan with a simple close(resume). The goroutine will wait the signal with something than :
for {
_, more := <-resume
if !more {
break
}
}
//update or not
Is there any other good idea to do this ? One of the above solutions is better than others ?
I'm not sure if I understand your question completely, but a simple solution to blocking the main thread is with an OS signal ie.:
done := make(chan os.Signal, 1)
signal.Notify(done, os.Interrupt, syscall.SIGTERM)
// Blocks main thread
<-done
This doesn't have to be an OS signal, this could be a channel that receives a struct{}{} value from somewhere else in your runtime.
If you need to do this as a result of multiple go routines working, I'd look into using sync.WaitGroup.
OK. Thanks to #blixenkrone I found an elegant solution using WaitGroup !
Usually, we use a sync.WaitGroup to synchronize goroutines and be sure that they finish their job : you add "1" to the WaitGroup and then wait... Goroutines, when they finish their job, use the Done() function to decrement the Waitgroup counter. In fact, the Wait() function of the WaitGroup must act like a for loop, looking for its internal counter to become "0" to resume.
What I do is exactly the opposite !!! I create a WaitGroup and increment it to 1 in the main program before launching the goroutines, and pass it to them. The goroutine do their first past of job, notify the main program and then wait for the WaitGroup. When the main program get every notification of first part job, it release the waitgroup, then each goroutine can finish the job. Here is a squeleton of what I did :
func main() {
structsToUpdate := getStructs() //some func to get my structs
communication := make(chan bool)
//the waitgroup used to block the goroutine
var resume sync.WaitGroup
resume.Add(1) //The waitgroup will be blocked !!!
success := 0
for _,structToUpdate := range structsToUpdates {
go updateStruct(&structToUpdate,communication,&resume)
}
//waiting for all goroutines do their first part of job
for i:=0; i<len(structsToUpdates); i++ {
res := <- communication
if res {
success++
}
}
//all structs should have done with first part now... Resume their job !
resume.Done()
//receive second part job
for success != 0 {
<-communication
success--
}
}
func updateStruct(s *StructToUpdate, c chan bool, resume *sync.WaitGroup) {
//First part of the job
result, err := performSomeRequest(s)
if err != nil {
//something wrong happened
c <- false
return
}
informGlobalWatcherOfResult(result)
c <- true //I did my job
//wait for all goroutines do their first part job
resume.Wait()
//here the main program 'Done()' the WaitGroup so it resumes the execution of the goroutine
updateOrNotStruct() //Some operations...
c <- true
}
Even if I think the wg.Wait() func should call a for loop itself, I think this code is cleaner than the solutions a proposed above...
I have this snippet of code which concurrently runs a function using an input and output channel and associated WaitGroups, but I was clued in to the fact that I've done some things wrong. Here's the code:
func main() {
concurrency := 50
var tasksWG sync.WaitGroup
tasks := make(chan string)
output := make(chan string)
for i := 0; i < concurrency; i++ {
tasksWG.Add(1)
// evidentally because I'm processing tasks in a groutine then I'm not blocking and I end up closing the tasks channel almost immediately and stopping tasks from executing
go func() {
for t := range tasks {
output <- process(t)
continue
}
tasksWG.Done()
}()
}
var outputWG sync.WaitGroup
outputWG.Add(1)
go func() {
for o := range output {
fmt.Println(o)
}
outputWG.Done()
}()
go func() {
// because of what was mentioned in the previous comment, the tasks wait group finishes almost immediately which then closes the output channel almost immediately as well which ends ranging over output early
tasksWG.Wait()
close(output)
}()
f, err := os.Open(os.Args[1])
if err != nil {
log.Panic(err)
}
s := bufio.NewScanner(f)
for s.Scan() {
tasks <- s.Text()
}
close(tasks)
// and finally the output wait group finishes almost immediately as well because tasks gets closed right away due to my improper use of goroutines
outputWG.Wait()
}
func process(t string) string {
time.Sleep(3 * time.Second)
return t
}
I've indicated in the comments where I've implementing things wrong. Now these comments make sense to me. The funny thing is that this code does indeed seem to run asynchronously and dramatically speeds up execution. I want to understand what I've done wrong but it's hard to wrap my head around it when the code seems to execute in an asynchronous way. I'd love to understand this better.
Your main goroutine is doing a couple of things sequentially and others concurrently, so I think your order of execution is off
f, err := os.Open(os.Args[1])
if err != nil {
log.Panic(err)
}
s := bufio.NewScanner(f)
for s.Scan() {
tasks <- s.Text()
}
Shouldn't you move this up top? So then you have values sent to tasks
THEN have your loop which ranges over tasks 50 times in the concurrency named for loop (you want to have something in tasks before calling code that ranges over it)
go func() {
// because of what was mentioned in the previous comment, the tasks wait group finishes almost immediately which then closes the output channel almost immediately as well which ends ranging over output early
tasksWG.Wait()
close(output)
}()
The logic here is confusing me, you're spawning a goroutine to wait on the waitgroup, so here the wait is nonblocking on the main goroutine - is that what you want to do? It won't wait for tasksWG to be decremented to zero inside main, it'll do that inside the goroutine that you've created. I don't believe you want to do that?
It might be easier to debug if you could give more details on the expected output?
I am trying to write a function to execute multiple jobs, when all the jobs are done, I want the control to go after wg.Wait(). I have mentioned different problems, I am facing in comments in the code.
How can I make it work?
func (q *ChanExecutor) Perform(ctx context.Context, name string, taskData *interface{}) chan *job.JobResult {
var waitgroup sync.WaitGroup
waitgroup.Add(1)
go func(wg *sync.WaitGroup) {
for j := range q.jobCh { // This is the channel which gives jobs
wg.Add(1)
go func(qq *ChanExecutor, jVal job.Job) { // we are just passing these values to closure. Is this necessary?
jobResultChannel:= jVal.Do(ctx) // Here we are executing the job as result which sends another channel of results
donech := jVal.DoneCh() // Job returns another channel which tells if that job is done
for true {
select {
case res := <-jobResultChannel:
q.result <- res // From the result we are passing that result to another channel
case syncJobDone := <-donech:
if syncJobDone {
donech = nil // here if the donech receives true it should come out of the select and for loop and the goroutine. How to do that?
// Another thing here, if the donech returns true before jobResultChannel then it should still go to jobResultChannel's case block
// The jVal.Do(ctx) executes the job and returns channel but in my case before starting the forloop both channels has values and donech has true value
wg.Done()
break
}
}
}
}(q, *j)
}
}(&waitgroup)
go func(wg *sync.WaitGroup, qq *ChanExecutor) {
time.Sleep(200 * time.Millisecond) // Here is another blunder. If I don't sleep here, randomly, it goes after wg.Wait()
// even though all the jobs are not done.
wg.Done() // REmoving the one which was added immediately after creating wg instance.
wg.Wait()
fmt.Println("Wait finish")
qq.Done()
}(&waitgroup, q)
fmt.Printf("returning result channel not result")
return q.result
}
First, you should remove the sleep and wg.Done from the second goroutine. It is sometimes failing without the sleep because sometimes the first goroutine does not have a chance to add to the wg before the second one removes it.
Second, you're trying to terminate the goroutine, so do just that:
if syncJobDone {
wg.Done()
return
}
TL;DR: A typical case of all goroutines are asleep, deadlock! but can't figure it out
I'm parsing the Wiktionary XML dump to build a DB of words. I defer the parsing of each article's text to a goroutine hoping that it will speed up the process.
It's 7GB and is processed in under 2 minutes in my machine when doing it serially, but if I can take advantage of all cores, why not.
I'm new to threading in general, I'm getting a all goroutines are asleep, deadlock! error.
What's wrong here?
This may not be performant at all, as it uses an unbuffered channel, so all goroutines effectively end up executing serially, but my idea is to learn and understand threading and to benchmark how long it takes with different alternatives:
unbuffered channel
different sized buffered channel
only calling as many goroutines at a time as there are runtime.NumCPU()
The summary of my code in pseudocode:
while tag := xml.getNextTag() {
wg.Add(1)
go parseTagText(chan, wg, tag.text)
// consume a channel message if available
select {
case msg := <-chan:
// do something with msg
default:
}
}
// reading tags finished, wait for running goroutines, consume what's left on the channel
for msg := range chan {
// do something with msg
}
// Sometimes this point is never reached, I get a deadlock
wg.Wait()
----
func parseTagText(chan, wg, tag.text) {
defer wg.Done()
// parse tag.text
chan <- whatever // just inform that the text has been parsed
}
Complete code:
https://play.golang.org/p/0t2EqptJBXE
In your complete example on the Go Playground, you:
Create a channel (line 39, results := make(chan langs)) and a wait-group (line 40, var wait sync.WaitGroup). So far so good.
Loop: in the loop, sometimes spin off a task:
if ...various conditions... {
wait.Add(1)
go parseTerm(results, &wait, text)
}
In the loop, sometimes do a non-blocking read from the channel (as shown in your question). No problem here either. But...
At the end of the loop, use:
for res := range results {
...
}
without ever calling close(results) in exactly one place, after all writers finish. This loop uses a blocking read from the channel. As long as some writer goroutine is still running, the blocking read can block without having the whole system stop, but when the last writer finishes writing and exits, there are no remaining writer goroutines. Any other remaining goroutines might rescue you, but there are none.
Since you use the var wait correctly (adding 1 in the right place, and calling Done() in the right place in the writer), the solution is to add one more goroutine, which will be the one to rescue you:
go func() {
wait.Wait()
close(results)
}()
You should spin off this rescuer goroutine just before entering the for res := range results loop. (If you spin it off any earlier, it might see the wait variable count down to zero too soon, just before it gets counted up again by spinning off another parseTerm.)
This anonymous function will block in the wait variable's Wait() function until the last writer goroutine has called the final wait.Done(), which will unblock this goroutine. Then this goroutine will call close(results), which will arrange for the for loop in your main goroutine to finish, unblocking that goroutine. When this goroutine (the rescuer) returns and thus terminates, there are no more rescuers, but we no longer need any.
(This main code then calls wait.Wait() unnecessarily: Since the for didn't terminate until the wait.Wait() in the new goroutine already unblocked, we know that this next wait.Wait() will return immediately. So we can drop this second call, although leaving it in is harmless.)
The problem is that nothing is closing the results channel, yet the range loop only exits when it closes. I've simplified your code to illustrate this and propsed a solution - basically consume the data in a goroutine:
// This is our producer
func foo(i int, ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
ch <- i
fmt.Println(i, "done")
}
// This is our consumer - it uses a different WG to signal it's done
func consumeData(ch chan int, wg *sync.WaitGroup) {
defer wg.Done()
for x := range ch {
fmt.Println(x)
}
fmt.Println("ALL DONE")
}
func main() {
ch := make(chan int)
wg := sync.WaitGroup{}
// create the producers
for i := 0; i < 10; i++ {
wg.Add(1)
go foo(i, ch, &wg)
}
// create the consumer on a different goroutine, and sync using another WG
consumeWg := sync.WaitGroup{}
consumeWg.Add(1)
go consumeData(ch,&consumeWg)
wg.Wait() // <<<< means that the producers are done
close(ch) // << Signal the consumer to exit
consumeWg.Wait() // << Wait for the consumer to exit
}
func GoCountColumns(in chan []string, r chan Result, quit chan int) {
for {
select {
case data := <-in:
r <- countColumns(data) // some calculation function
case <-quit:
return // stop goroutine
}
}
}
func main() {
fmt.Println("Welcome to the csv Calculator")
file_path := os.Args[1]
fd, _ := os.Open(file_path)
reader := csv.NewReader(bufio.NewReader(fd))
var totalColumnsCount int64 = 0
var totallettersCount int64 = 0
linesCount := 0
numWorkers := 10000
rc := make(chan Result, numWorkers)
in := make(chan []string, numWorkers)
quit := make(chan int)
t1 := time.Now()
for i := 0; i < numWorkers; i++ {
go GoCountColumns(in, rc, quit)
}
//start worksers
go func() {
for {
record, err := reader.Read()
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
if linesCount%1000000 == 0 {
fmt.Println("Adding to the channel")
}
in <- record
//data := countColumns(record)
linesCount++
//totalColumnsCount = totalColumnsCount + data.ColumnCount
//totallettersCount = totallettersCount + data.LettersCount
}
close(in)
}()
for i := 0; i < numWorkers; i++ {
quit <- 1 // quit goroutines from main
}
close(rc)
for i := 0; i < linesCount; i++ {
data := <-rc
totalColumnsCount = totalColumnsCount + data.ColumnCount
totallettersCount = totallettersCount + data.LettersCount
}
fmt.Printf("I counted %d lines\n", linesCount)
fmt.Printf("I counted %d columns\n", totalColumnsCount)
fmt.Printf("I counted %d letters\n", totallettersCount)
elapsed := time.Now().Sub(t1)
fmt.Printf("It took %f seconds\n", elapsed.Seconds())
}
My Hello World is a program that reads a csv file and passes it to a channel. Then the goroutines should consume from this channel.
My Problem is I have no idea how to detect from the main thread that all data was processed and I can exit my program.
on top of other answers.
Take (great) care that closing a channel should happen on the write call site, not the read call site. In GoCountColumns the r channel being written, the responsibility to close the channel are onto GoCountColumns function. Technical reasons are, it is the only actor knowing for sure that the channel will not being written anymore and thus is safe for close.
func GoCountColumns(in chan []string, r chan Result, quit chan int) {
defer close(r) // this line.
for {
select {
case data := <-in:
r <- countColumns(data) // some calculation function
case <-quit:
return // stop goroutine
}
}
}
The function parameters naming convention, if i might say, is to have the destination as first parameter, the source as second, and others parameters along. The GoCountColumns is preferably written:
func GoCountColumns(dst chan Result, src chan []string, quit chan int) {
defer close(dst)
for {
select {
case data := <-src:
dst <- countColumns(data) // some calculation function
case <-quit:
return // stop goroutine
}
}
}
You are calling quit right after the process started. Its illogical. This quit command is a force exit sequence, it should be called once an exit signal is detected, to force exit the current processing in best state possible, possibly all broken. In other words, you should be relying on the signal.Notify package to capture exit events, and notify your workers to quit. see https://golang.org/pkg/os/signal/#example_Notify
To write better parallel code, list at first the routines you need to manage the program lifetime, identify those you need to block onto to ensure the program has finished before exiting.
In your code, exists read, map. To ensure complete processing, the program main function must ensure that it captures a signal when map exits before exiting itself. Notice that the read function does not matter.
Then, you will also need the code required to capture an exit event from user input.
Overall, it appears we need to block onto two events to manage lifetime. Schematically,
func main(){
go read()
go map(mapDone)
go signal()
select {
case <-mapDone:
case <-sig:
}
}
This simple code is good to process or die. Indeed, when the user event is caught, the program exits immediately, without giving a chance to others routines to do something required upon stop.
To improve those behaviors, you need first a way to signal the program wants to leave to other routines, second, a way to wait for those routines to finish their stop sequence before leaving.
To signal exit event, or cancellation, you can make use of a context.Context, pass it around to the workers, make them listen to it.
Again, schematically,
func main(){
ctx,cancel := context.WithCancel(context.WithBackground())
go read(ctx)
go map(ctx,mapDone)
go signal()
select {
case <-mapDone:
case <-sig:
cancel()
}
}
(more onto read and map later)
To wait for completion, many things are possible, for as long as they are thread safe. Usually, a sync.WaitGroup is being used. Or, in cases like yours where there is only one routine to wait for, we can re use the current mapDone channel.
func main(){
ctx,cancel := context.WithCancel(context.WithBackground())
go read(ctx)
go map(ctx,mapDone)
go signal()
select {
case <-mapDone:
case <-sig:
cancel()
<-mapDone
}
}
That is simple and straight forward. But it is not totally correct. The last mapDone chan might block forever and make the program unstoppable. So you might implement a second signal handler, or a timeout.
Schematically, the timeout solution is
func main(){
ctx,cancel := context.WithCancel(context.WithBackground())
go read(ctx)
go map(ctx,mapDone)
go signal()
select {
case <-mapDone:
case <-sig:
cancel()
select {
case <-mapDone:
case <-time.After(time.Second):
}
}
}
You might also accumulate a signal handling and a timeout in the last select.
Finally, there are few things to tell about read and map context listening.
Starting with map, the implementation requires to read for context.Done channel regularly to detect cancellation.
It is the easy part, it requires to only update the select statement.
func GoCountColumns(ctx context.Context, dst chan Result, src chan []string) {
defer close(dst)
for {
select {
case <-ctx.Done():
<-time.After(time.Minute) // do something more useful.
return // quit. Notice the defer will be called.
case data := <-src:
dst <- countColumns(data) // some calculation function
}
}
}
Now the read part is bit more tricky as it is an IO it does not provide a selectable programming interface and listening to the context channel cancellation might seem contradictory. It is. As IOs are blocking, impossible to listen the context. And while reading from the context channel, impossible to read the IO. In your case, the solution requires to understand that your read loop is not relevant to your program lifetime (recall we only listen onto mapDone?), and that we can just ignore the context.
In other cases, if for example you wanted to restart at last byte read (so at every read, we increment an n, counting bytes, and we want to save that value upon stop). Then, a new routine is required to be started, and thus, multiple routines are to wait for completion. In such cases a sync.WaitGroup will be more appropriate.
Schematically,
func main(){
var wg sync.WaitGroup
processDone:=make(chan struct{})
ctx,cancel := context.WithCancel(context.WithBackground())
go read(ctx)
wg.Add(1)
go saveN(ctx,&wg)
wg.Add(1)
go map(ctx,&wg)
go signal()
go func(){
wg.Wait()
close(processDone)
}()
select {
case <-processDone:
case <-sig:
cancel()
select {
case <-processDone:
case <-time.After(time.Second):
}
}
}
In this last code, the waitgroup is being passed around. Routines are responsible to call for wg.Done(), when all routines are done, the processDone channel is closed, to signal the select.
func GoCountColumns(ctx context.Context, dst chan Result, src chan []string, wg *sync.WaitGroup) {
defer wg.Done()
defer close(dst)
for {
select {
case <-ctx.Done():
<-time.After(time.Minute) // do something more useful.
return // quit. Notice the defer will be called.
case data := <-src:
dst <- countColumns(data) // some calculation function
}
}
}
It is undecided which patterns is preferred, but you might also see waitgroup being managed at call sites only.
func main(){
var wg sync.WaitGroup
processDone:=make(chan struct{})
ctx,cancel := context.WithCancel(context.WithBackground())
go read(ctx)
wg.Add(1)
go func(){
defer wg.Done()
saveN(ctx)
}()
wg.Add(1)
go func(){
defer wg.Done()
map(ctx)
}()
go signal()
go func(){
wg.Wait()
close(processDone)
}()
select {
case <-processDone:
case <-sig:
cancel()
select {
case <-processDone:
case <-time.After(time.Second):
}
}
}
Beyond all of that and OP questions, you must always evaluate upfront the pertinence of parallel processing for a given task. There is no unique recipe, practice and measure your code performances. see pprof.
There is way too much going on in this code. You should restructure your code into short functions that serve specific purposes to make it possible for someone to help you out easily (and help yourself as well).
You should read the following Go article, which goes into concurrency patterns:
https://blog.golang.org/pipelines
There are multiple ways to make one go-routine wait on some other work to finish. The most common ways are with wait groups (example I have provided) or channels.
func processSomething(...) {
...
}
func main() {
workers := &sync.WaitGroup{}
for i := 0; i < numWorkers; i++ {
workers.Add(1) // you want to call this from the calling go-routine and before spawning the worker go-routine
go func() {
defer workers.Done() // you want to call this from the worker go-routine when the work is done (NOTE the defer, which ensures it is called no matter what)
processSomething(....) // your async processing
}()
}
// this will block until all workers have finished their work
workers.Wait()
}
You can use a channel to block main until completion of a goroutine.
package main
import (
"log"
"time"
)
func main() {
c := make(chan struct{})
go func() {
time.Sleep(3 * time.Second)
log.Println("bye")
close(c)
}()
// This blocks until the channel is closed by the routine
<-c
}
No need to write anything into the channel. Reading is blocked until data is read or, which we use here, the channel is closed.