this is code
func main() {
...
pool := createPool(*redis_server, *redis_pass)
defer pool.Close()
c := pool.Get()
var i int64
st := tickSec()
for i = 0; i < *total; i++ {
r := time.Now().Unix() - rand.Int63n(60*60*24*31*12)
score, _ := strconv.Atoi(time.Unix(r, 0).Format("2006010215"))
id := utee.PlainMd5(uuid.NewUUID().String())
c.Send("ZADD", "app_a_5512", score, id)
if i%10000 == 0 {
c.Flush()
log.Println("current sync to redis", i)
}
}
//c.Flush()
c.Close()
...
}
if i use c.Close(),the total set 100000,the real sortedset count 100000.
but if i use c.Flush(),the total also set 100000, the real sortedset count less than 100000(96932);if i use time.Sleep() in the end of the main func,the total is 100000 too.
when main func exit,the flush func is not complete?and why? thank you!
The reason that the program works when Close() is called after the loop is that the pooled connection's Close() method reads and discards all pending responses.
The application should Receive the responses for all commands instead of letting the respones backup and consume memory on the server. There's no need to flush in the loop.
go func() {
for i = 0; i < *total; i++ {
r := time.Now().Unix() - rand.Int63n(60*60*24*31*12)
score, _ := strconv.Atoi(time.Unix(r, 0).Format("2006010215"))
id := utee.PlainMd5(uuid.NewUUID().String())
c.Send("ZADD", "app_a_5512", score, id)
}
c.Flush()
}
for i = 0; i < *total; i++ {
c.Receive()
}
c.Close()
Also, the application should check and handle the errors returns from Send, Flush and Receive.
Related
I'm writing an app using Go that is interacting with Spotify's API and I find myself needing to use an infinite for loop to call an endpoint until the length of the returned slice is less than the limit, signalling that I've reached the end of the available entries.
For my user account, there are 1644 saved albums (I determined this by looping through without using goroutines). However, when I add goroutines in, I'm getting back 2544 saved albums with duplicates. I'm also using the semaphore pattern to limit the number of goroutines so that I don't exceed the rate limit.
I assume that the issue is with using the active variable rather than channels, but my attempt at that just resulted in an infinite loop
wg := &sync.WaitGroup{}
sem := make(chan bool, 20)
active := true
offset := 0
for {
sem <- true
if active {
// add each new goroutine to waitgroup
wg.Add(1)
go func() error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
// release the worker
defer func() { <-sem }()
savedAlbums, err := client.CurrentUsersAlbums(ctx, spotify.Limit(50), spotify.Offset(offset))
if err != nil {
return err
}
userAlbums = append(userAlbums, savedAlbums.Albums...)
if len(savedAlbums.Albums) < 50 {
// since the limit is set to 50, we know that if the number of returned albums
// is less than 50 that we're done retrieving data
active = false
return nil
} else {
offset += 50
return nil
}
}()
} else {
wg.Wait()
break
}
}
Thanks in advance!
I suspect that your main issue may be a misunderstanding of what the go keyword does; from the docs:
A "go" statement starts the execution of a function call as an independent concurrent thread of control, or goroutine, within the same address space.
So go func() error { starts the execution of the closure; it does not mean that any of the code runs immediately. In fact because, client.CurrentUsersAlbums will take a while, it's likely you will be requesting the first 50 items 20 times. This can be demonstrated with a simplified version of your application (playground)
func main() {
wg := &sync.WaitGroup{}
sem := make(chan bool, 20)
active := true
offset := 0
for {
sem <- true
if active {
// add each new goroutine to waitgroup
wg.Add(1)
go func() error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
// release the worker
defer func() { <-sem }()
fmt.Println("Getting from:", offset)
time.Sleep(time.Millisecond) // Simulate the query
// Pretend that we got back 50 albums
offset += 50
if offset > 2000 {
active = false
}
return nil
}()
} else {
wg.Wait()
break
}
}
}
Running this will produce somewhat unpredictable results (note that the playground caches results so try it on your machine) but you will probably see 20 X Getting from: 0.
A further issue is data races. Updating a variable from multiple goroutines without protection (e.g. sync.Mutex) results in undefined behaviour.
You will want to know how to fix this but unfortunately you will need to rethink your algorithm. Currently the process you are following is:
Set pos to 0
Get 50 records starting from pos
If we got 50 records then pos=pos+50 and loop back to step 2
This is a sequential algorithm; you don't know whether you have all of the data until you have requested the previous section. I guess you could make speculative queries (and handle failures) but a better solution would be to find some way to determine the number of results expected and then split the queries to get that number of records between multiple goroutines.
Note that if you do know the number of responses then you can do something like the following (playground):
noOfResultsToGet := 1644 // In the below we are getting 0-1643
noOfResultsPerRequest := 50
noOfSimultaneousRequests := 20 // You may not need this but many services will limit the number of simultaneous requests you can make (or, at least, rate limit them)
requestChan := make(chan int) // Will be passed the starting #
responseChan := make(chan []string) // Response from whatever request we are making (can be any type really)
// Start goroutines to make the requests
var wg sync.WaitGroup
wg.Add(noOfSimultaneousRequests)
for i := 0; i < noOfSimultaneousRequests; i++ {
go func(routineNo int) {
defer wg.Done()
for startPos := range requestChan {
// Simulate making the request
maxResult := startPos + noOfResultsPerRequest
if maxResult > noOfResultsToGet {
maxResult = noOfResultsToGet
}
rsp := make([]string, 0, noOfResultsPerRequest)
for x := startPos; x < maxResult; x++ {
rsp = append(rsp, strconv.Itoa(x))
}
responseChan <- rsp
fmt.Printf("Goroutine %d handling data from %d to %d\n", routineNo, startPos, startPos+noOfResultsPerRequest)
}
}(i)
}
// Close the response channel when all goroutines have shut down
go func() {
wg.Wait()
close(responseChan)
}()
// Send the requests
go func() {
for reqFrom := 0; reqFrom < noOfResultsToGet; reqFrom += noOfResultsPerRequest {
requestChan <- reqFrom
}
close(requestChan) // Allow goroutines to exit
}()
// Receive responses (note that these may be out of order)
result := make([]string, 0, noOfResultsToGet)
for x := range responseChan {
result = append(result, x...)
}
// Order the results and output (results from gorouting may come back in any order)
sort.Slice(result, func(i, j int) bool {
a, _ := strconv.Atoi(result[i])
b, _ := strconv.Atoi(result[j])
return a < b
})
fmt.Printf("Result: %v", result)
Relying on channels to pass messages often makes this kind of thing easier to think about and reduces the chance that you will make a mistake.
Set offset as an args -> go func(offset int) error {.
Increment offset by 50 after calling go func
Change active type to chan bool
To avoid data race on userAlbums = append(userAlbums, res...). We need to create channel that same type as userAlbums, then run for loop inside goroutine, then send the results to that channel.
this is the example : https://go.dev/play/p/yzk8qCURZFC
if applied to your code :
wg := &sync.WaitGroup{}
worker := 20
active := make(chan bool, worker)
for i := 0; i < worker; i++ {
active <- true
}
// I assume the type of userAlbums is []string
resultsChan := make(chan []string, worker)
go func() {
offset := 0
for {
if <-active {
// add each new goroutine to waitgroup
wg.Add(1)
go func(offset int) error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
savedAlbums, err := client.CurrentUsersAlbums(ctx, spotify.Limit(50), spotify.Offset(offset))
if err != nil {
// active <- false // maybe you need this
return err
}
resultsChan <- savedAlbums.Albums
if len(savedAlbums.Albums) < 50 {
// since the limit is set to 50, we know that if the number of returned albums
// is less than 50 that we're done retrieving data
active <- false
return nil
} else {
active <- true
return nil
}
}(offset)
offset += 50
} else {
wg.Wait()
close(resultsChan)
break
}
}
}()
for res := range resultsChan {
userAlbums = append(userAlbums, res...)
}
I have the following piece of code. I'm trying to run 3 GO routines at the same time never exceeding three. This works as expected, but the code is supposed to be running updates a table in the DB.
So the first routine processes the first 50, then the second 50, and then third 50, and it repeats. I don't want two routines processing the same rows at the same time and due to how long the update takes, this happens almost every time.
To solve this, I started flagging the rows with a new column processing which is a bool. I set it to true for all rows to be updated when the routine starts and sleep the script for 6 seconds to allow the flag to be updated.
This works for a random amount of time, but every now and then, I'll see 2-3 jobs processing the same rows again. I feel like the method I'm using to prevent duplicate updates is a bit janky and was wondering if there was a better way.
stopper := make(chan struct{}, 3)
var counter int
for {
counter++
stopper <- struct{}{}
go func(db *sqlx.DB, c int) {
fmt.Println("start")
updateTables(db)
fmt.Println("stop"b)
<-stopper
}(db, counter)
time.Sleep(6 * time.Second)
}
in updateTables
var ids[]string
err := sqlx.Select(db, &data, `select * from table_data where processing = false `)
if err != nil {
panic(err)
}
for _, row:= range data{
list = append(ids, row.Id)
}
if len(rows) == 0 {
return
}
for _, row:= range data{
_, err = db.Exec(`update table_data set processing = true where id = $1, row.Id)
if err != nil {
panic(err)
}
}
// Additional row processing
I think there's a misunderstanding on approach to go routines in this case.
Go routines to do these kind of work should be approached like worker Threads, using channels as the communication method in between the main routine (which will be doing the synchronization) and the worker go routines (which will be doing the actual job).
package main
import (
"log"
"sync"
"time"
)
type record struct {
id int
}
func main() {
const WORKER_COUNT = 10
recordschan := make(chan record)
var wg sync.WaitGroup
for k := 0; k < WORKER_COUNT; k++ {
wg.Add(1)
// Create the worker which will be doing the updates
go func(workerID int) {
defer wg.Done() // Marking the worker as done
for record := range recordschan {
updateRecord(record)
log.Printf("req %d processed by worker %d", record.id, workerID)
}
}(k)
}
// Feeding the records channel
for _, record := range fetchRecords() {
recordschan <- record
}
// Closing our channel as we're not using it anymore
close(recordschan)
// Waiting for all the go routines to finish
wg.Wait()
log.Println("we're done!")
}
func fetchRecords() []record {
result := []record{}
for k := 0; k < 100; k++ {
result = append(result, record{k})
}
return result
}
func updateRecord(req record) {
time.Sleep(200 * time.Millisecond)
}
You can even buffer things in the main go routine if you need to update all the 50 tables at once.
I am trying to design a HTTP client in Go that will be capable ofcConcurrent API calls to the web services and write some data in a textfile.
func getTotalCalls() int {
reader := bufio.NewReader(os.Stdin)
...
return callInt
}
getTotalColls decide how many calls I want to make, input comes from terminal.
func writeToFile(s string, namePrefix string) {
fileStore := fmt.Sprintf("./data/%s_calls.log", namePrefix)
...
defer f.Close()
if _, err := f.WriteString(s); err != nil {
log.Println(err)
}
}
The writeToFile will write data to file synchronously from a buffered channel.
func makeRequest(url string, ch chan<- string, id int) {
var jsonStr = []byte(`{"from": "Saru", "message": "Saru to Discovery. Over!"}`)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
start := time.Now()
resp, err := client.Do(req)
if err != nil {
panic(err)
}
secs := time.Since(start).Seconds()
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
ch <- fmt.Sprintf("%d, %.2f, %d, %s, %s\n", id, secs, len(body), url, body)
}
This is the function which make the API call in a go Routine.
and Finally Here is the Main function, which send data from go routine to a bufferend channel and Later I range over the bufferend channel of string and write the data to file.
func main() {
urlPrefix := os.Getenv("STARCOMM_GO")
url := urlPrefix + "discovery"
totalCalls := getTotalCalls()
queue := make(chan string, totalCalls)
for i := 1; i <= totalCalls; i++ {
go makeRequest(url, queue, i)
}
for item := range queue {
fmt.Println(item)
writeToFile(item, fmt.Sprint(totalCalls))
}
}
The problem is at the end of the call the buffered somehow block and the program wait forever end of all the call. Does someone have a better way to design such use case? My final goal is to check for different number of concurrent post request how much time it takes for each calls for bench marking the API endpoint for 5, 10, 50, 100, 500, 1000 ... set of concurrent call.
Something has to close(queue). Otherwise range queue will block. If you want to range queue, you have to ensure that this channel is closed once the final client is done.
However... It's not even clear that you need to range queue though, since you know exactly how many results you'll get - it's totalCalls. You just need to loop this many times receiving from queue.
I believe your use case is similar to the Worker Pools example on gobyexample, so you may want to check that one out. Here's the code from that example:
// In this example we'll look at how to implement
// a _worker pool_ using goroutines and channels.
package main
import (
"fmt"
"time"
)
// Here's the worker, of which we'll run several
// concurrent instances. These workers will receive
// work on the `jobs` channel and send the corresponding
// results on `results`. We'll sleep a second per job to
// simulate an expensive task.
func worker(id int, jobs <-chan int, results chan<- int) {
for j := range jobs {
fmt.Println("worker", id, "started job", j)
time.Sleep(time.Second)
fmt.Println("worker", id, "finished job", j)
results <- j * 2
}
}
func main() {
// In order to use our pool of workers we need to send
// them work and collect their results. We make 2
// channels for this.
const numJobs = 5
jobs := make(chan int, numJobs)
results := make(chan int, numJobs)
// This starts up 3 workers, initially blocked
// because there are no jobs yet.
for w := 1; w <= 3; w++ {
go worker(w, jobs, results)
}
// Here we send 5 `jobs` and then `close` that
// channel to indicate that's all the work we have.
for j := 1; j <= numJobs; j++ {
jobs <- j
}
close(jobs)
// Finally we collect all the results of the work.
// This also ensures that the worker goroutines have
// finished. An alternative way to wait for multiple
// goroutines is to use a [WaitGroup](waitgroups).
for a := 1; a <= numJobs; a++ {
<-results
}
}
Your "worker" makes HTTP requests, otherwise it's pretty much the same pattern. Note the for loop at the end which reads from the channel a known number of times.
If you need to limit a number of simultaneous requests, you can use a semaphore implemented with a buffered channel.
func makeRequest(url string, id int) string {
var jsonStr = []byte(`{"from": "Saru", "message": "Saru to Discovery. Over!"}`)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
start := time.Now()
resp, err := client.Do(req)
if err != nil {
panic(err)
}
secs := time.Since(start).Seconds()
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
return fmt.Sprintf("%d, %.2f, %d, %s, %s\n", id, secs, len(body), url, body)
}
func main() {
urlPrefix := os.Getenv("STARCOMM_GO")
url := urlPrefix + "discovery"
totalCalls := getTotalCalls()
concurrencyLimit := 50 // 5, 10, 50, 100, 500, 1000.
// Declare semaphore as a buffered channel with capacity limited by concurrency level.
semaphore := make(chan struct{}, concurrencyLimit)
for i := 1; i <= totalCalls; i++ {
// Take a slot in semaphore before proceeding.
// Once all slots are taken this call will block until slot is freed.
semaphore <- struct{}{}
go func() {
// Release slot on job finish.
defer func() { <-semaphore }()
item := makeRequest(url, i)
fmt.Println(item)
// Beware that writeToFile will be called concurrently and may need some synchronization.
writeToFile(item, fmt.Sprint(totalCalls))
}()
}
// Wait for jobs to finish by filling semaphore to full capacity.
for i := 0; i < cap(semaphore); i++ {
semaphore <- struct{}{}
}
close(semaphore)
}
I'm tying to execute things async with multiple go routines. I pass in the number of "threads" to use to process the files async. The files is an array of Strings to process.
queue := make(chan string)
threadCount := c.Int("threads")
if c.Int("threads") < len(files) {
threadCount = len(files)
}
log.Infof("Starting %i processes", c.Int("threads"))
for i := 0; i < threadCount; i++ {
go renderGoRoutine(queue)
}
for _, f := range files {
queue <- f
}
close(queue)
And the routine itself looks like this:
func renderGoRoutine(queue chan string) {
for file := range queue {
// do some heavy lifting stuff with the file
}
}
This does work fine whenever i use just one thread. As soon as i take more then one it does exit/leave the scope before it is done with all the go routines.
How do I make it process everything?
Previous question: Using a channel for dispatching tasks to go routine
Using WaitGroups is an option.
At the beginning, you add number tasks into WaitGroup and after each task is done decrement counter in the WaitGroup. Wait until all tasks are finished at the end of your code flow.
See the example: https://godoc.org/sync#WaitGroup
Your code will look like this:
queue := make(chan string)
wg := sync.WaitGroup{}
wg.Add(len(files))
threadCount := c.Int("threads")
if c.Int("threads") < len(files) {
threadCount = len(files)
}
log.Infof("Starting %i processes", c.Int("threads"))
for i := 0; i < threadCount; i++ {
go renderGoRoutine(queue)
}
for _, f := range files {
queue <- f
}
close(queue)
wg.Wait()
renderGoRoutine:
func renderGoRoutine(queue chan string) {
for file := range queue {
// do some heavy lifting stuff with the file
// decrement the waitGroup counter
wg.Done()
}
}
You are using the channel to publish work to be done. As soon as the last item is taken from the queue (not finished processing), your program exits.
You could use a channel to write to at the end of renderGoRoutine to signal the end of processing.
At the top:
sync := make(chan bool)
In renderGoRoutine at the end (Assuming it is in the same file):
sync <- true
At the bottom:
for f := range sync {
<- sync
}
Now your program waits until the number of files are processed.
Or to have a complete example:
queue := make(chan string)
sync := make(chan bool)
threadCount := c.Int("threads")
if c.Int("threads") < len(files) {
threadCount = len(files)
}
log.Infof("Starting %i processes", c.Int("threads"))
for i := 0; i < threadCount; i++ {
go renderGoRoutine(queue)
}
for _, f := range files {
queue <- f
}
close(queue)
for f := range sync {
<- sync
}
And the routine should be changed like this:
func renderGoRoutine(queue chan string) {
for file := range queue {
// do some heavy lifting stuff with the file
sync <- true
}
}
I did forget to wait for all tasks to finish. This can simply be done by waiting for all loops to end. Since close(channel) does end the for range channel a simple sync with a channel can be used like so:
sync := make(chan bool)
queue := make(chan string)
threadCount := c.Int("threads")
if c.Int("threads") < len(files) {
threadCount = len(files)
}
log.Infof("Starting %i processes", c.Int("threads"))
for i := 0; i < threadCount; i++ {
go renderGoRoutine(queue)
}
for _, f := range files {
queue <- f
}
close(queue)
for i := 0; i < threadCount; i++ {
<- sync
}
And last but not least write to the channel whenever a iteration is stopped.
func renderGoRoutine(queue chan string) {
for file := range queue { //whatever is done here
}
sync <- true
}
I see lots of tutorials and examples on how to make Go wait for x number of goroutines to finish, but what I'm trying to do is have ensure there are always x number running, so a new goroutine is launched as soon as one ends.
Specifically I have a few hundred thousand 'things to do' which is processing some stuff that is coming out of MySQL. So it works like this:
db, err := sql.Open("mysql", connection_string)
checkErr(err)
defer db.Close()
rows,err := db.Query(`SELECT id FROM table`)
checkErr(err)
defer rows.Close()
var id uint
for rows.Next() {
err := rows.Scan(&id)
checkErr(err)
go processTheThing(id)
}
checkErr(err)
rows.Close()
Currently that will launch several hundred thousand threads of processTheThing(). What I need is that a maximum of x number (we'll call it 20) goroutines are launched. So it starts by launching 20 for the first 20 rows, and from then on it will launch a new goroutine for the next id the moment that one of the current goroutines has finished. So at any point in time there are always 20 running.
I'm sure this is quite simple/standard, but I can't seem to find a good explanation on any of the tutorials or examples or how this is done.
You may find Go Concurrency Patterns article interesting, especially Bounded parallelism section, it explains the exact pattern you need.
You can use channel of empty structs as a limiting guard to control number of concurrent worker goroutines:
package main
import "fmt"
func main() {
maxGoroutines := 10
guard := make(chan struct{}, maxGoroutines)
for i := 0; i < 30; i++ {
guard <- struct{}{} // would block if guard channel is already filled
go func(n int) {
worker(n)
<-guard
}(i)
}
}
func worker(i int) { fmt.Println("doing work on", i) }
Here I think something simple like this will work :
package main
import "fmt"
const MAX = 20
func main() {
sem := make(chan int, MAX)
for {
sem <- 1 // will block if there is MAX ints in sem
go func() {
fmt.Println("hello again, world")
<-sem // removes an int from sem, allowing another to proceed
}()
}
}
Thanks to everyone for helping me out with this. However, I don't feel that anyone really provided something that both worked and was simple/understandable, although you did all help me understand the technique.
What I have done in the end is I think much more understandable and practical as an answer to my specific question, so I will post it here in case anyone else has the same question.
Somehow this ended up looking a lot like what OneOfOne posted, which is great because now I understand that. But OneOfOne's code I found very difficult to understand at first because of the passing functions to functions made it quite confusing to understand what bit was for what. I think this way makes a lot more sense:
package main
import (
"fmt"
"sync"
)
const xthreads = 5 // Total number of threads to use, excluding the main() thread
func doSomething(a int) {
fmt.Println("My job is",a)
return
}
func main() {
var ch = make(chan int, 50) // This number 50 can be anything as long as it's larger than xthreads
var wg sync.WaitGroup
// This starts xthreads number of goroutines that wait for something to do
wg.Add(xthreads)
for i:=0; i<xthreads; i++ {
go func() {
for {
a, ok := <-ch
if !ok { // if there is nothing to do and the channel has been closed then end the goroutine
wg.Done()
return
}
doSomething(a) // do the thing
}
}()
}
// Now the jobs can be added to the channel, which is used as a queue
for i:=0; i<50; i++ {
ch <- i // add i to the queue
}
close(ch) // This tells the goroutines there's nothing else to do
wg.Wait() // Wait for the threads to finish
}
Create channel for passing data to goroutines.
Start 20 goroutines that processes the data from channel in a loop.
Send the data to the channel instead of starting a new goroutine.
Grzegorz Żur's answer is the most efficient way to do it, but for a newcomer it could be hard to implement without reading code, so here's a very simple implementation:
type idProcessor func(id uint)
func SpawnStuff(limit uint, proc idProcessor) chan<- uint {
ch := make(chan uint)
for i := uint(0); i < limit; i++ {
go func() {
for {
id, ok := <-ch
if !ok {
return
}
proc(id)
}
}()
}
return ch
}
func main() {
runtime.GOMAXPROCS(4)
var wg sync.WaitGroup //this is just for the demo, otherwise main will return
fn := func(id uint) {
fmt.Println(id)
wg.Done()
}
wg.Add(1000)
ch := SpawnStuff(10, fn)
for i := uint(0); i < 1000; i++ {
ch <- i
}
close(ch) //should do this to make all the goroutines exit gracefully
wg.Wait()
}
playground
This is a simple producer-consumer problem, which in Go can be easily solved using channels to buffer the paquets.
To put it simple: create a channel that accept your IDs. Run a number of routines which will read from the channel in a loop then process the ID. Then run your loop that will feed IDs to the channel.
Example:
func producer() {
var buffer = make(chan uint)
for i := 0; i < 20; i++ {
go consumer(buffer)
}
for _, id := range IDs {
buffer <- id
}
}
func consumer(buffer chan uint) {
for {
id := <- buffer
// Do your things here
}
}
Things to know:
Unbuffered channels are blocking: if the item wrote into the channel isn't accepted, the routine feeding the item will block until it is
My example lack a closing mechanism: you must find a way to make the producer to wait for all consumers to end their loop before returning. The simplest way to do this is with another channel. I let you think about it.
I've wrote a simple package to handle concurrency for Golang. This package will help you limit the number of goroutines that are allowed to run concurrently:
https://github.com/zenthangplus/goccm
Example:
package main
import (
"fmt"
"goccm"
"time"
)
func main() {
// Limit 3 goroutines to run concurrently.
c := goccm.New(3)
for i := 1; i <= 10; i++ {
// This function have to call before any goroutine
c.Wait()
go func(i int) {
fmt.Printf("Job %d is running\n", i)
time.Sleep(2 * time.Second)
// This function have to when a goroutine has finished
// Or you can use `defer c.Done()` at the top of goroutine.
c.Done()
}(i)
}
// This function have to call to ensure all goroutines have finished
// after close the main program.
c.WaitAllDone()
}
Also can take a look here: https://github.com/LiangfengChen/goutil/blob/main/concurrent.go
The example can refer the test case.
func TestParallelCall(t *testing.T) {
format := "test:%d"
data := make(map[int]bool)
mutex := sync.Mutex{}
val, err := ParallelCall(1000, 10, func(pos int) (interface{}, error) {
mutex.Lock()
defer mutex.Unlock()
data[pos] = true
return pos, errors.New(fmt.Sprintf(format, pos))
})
for i := 0; i < 1000; i++ {
if _, ok := data[i]; !ok {
t.Errorf("TestParallelCall pos not found: %d", i)
}
if val[i] != i {
t.Errorf("TestParallelCall return value is not right (%d,%v)", i, val[i])
}
if err[i].Error() != fmt.Sprintf(format, i) {
t.Errorf("TestParallelCall error msg is not correct (%d,%v)", i, err[i])
}
}
}