How can I make this simple for loop break after exactly one 1s has passed since its execution?
var i int
for {
i++
}
By checking the elapsed time since the start:
var i int
for start := time.Now(); time.Since(start) < time.Second; {
i++
}
Or using a "timeout" channel, acquired by calling time.After(). Use select to check if time is up, but you must add a default branch so it will be a non-blocking check. If time is up, break from the loop. Also very important to use a label and break from the for loop, else break will just break from the select and it will be an endless loop.
loop:
for timeout := time.After(time.Second); ; {
select {
case <-timeout:
break loop
default:
}
i++
}
Note: If the loop body also performs communication operations (like send or receive), using a timeout channel may be the only viable option! (You can list the timeout check and the loop's communication op in the same select.)
We may rewrite the timeout channel solution to not use a label:
for stay, timeout := true, time.After(time.Second); stay; {
i++
select {
case <-timeout:
stay = false
default:
}
}
Optimization
I know your loop is just an example, but if the loop is doing just a tiny bit of work, it is not worth checking the timeout in every iteration. We may rewrite the first solution to check timeout e.g. in every 10 iterations like this:
var i int
for start := time.Now(); ; {
if i % 10 == 0 {
if time.Since(start) > time.Second {
break
}
}
i++
}
We may choose an iteration number which is a multiple of 2, and then we may use bitmasks which is supposed to be even faster than remainder check:
var i int
for start := time.Now(); ; {
if i&0x0f == 0 { // Check in every 16th iteration
if time.Since(start) > time.Second {
break
}
}
i++
}
We may also calculate the end time once (when the loop must end), and then you just have to compare the current time to this:
var i int
for end := time.Now().Add(time.Second); ; {
if i&0x0f == 0 { // Check in every 16th iteration
if time.Now().After(end) {
break
}
}
i++
}
I know the question is a bit old, but below might be useful for someone looking for similar scenario:
func keepCheckingSomething() (bool, error) {
timeout := time.NewTimer(10 * time.Second)
ticker := time.NewTicker(500 * time.Millisecond)
defer timeout.Stop()
defer ticker.Stop()
// Keep trying until we're timed out or get a result/error
for {
select {
// Got a timeout! fail with a timeout error
case <-timeout:
// maybe, check for one last time
ok, err := checkSomething()
if !ok {
return false, errors.New("timed out")
}
return ok, err
// Got a tick, we should check on checkSomething()
case <-ticker:
ok, err := checkSomething()
if err != nil {
// We may return, or ignore the error
return false, err
// checkSomething() done! let's return
} else if ok {
return true, nil
}
// checkSomething() isn't done yet, but it didn't fail either, let's try again
}
}
}
Related
I'm writing an app using Go that is interacting with Spotify's API and I find myself needing to use an infinite for loop to call an endpoint until the length of the returned slice is less than the limit, signalling that I've reached the end of the available entries.
For my user account, there are 1644 saved albums (I determined this by looping through without using goroutines). However, when I add goroutines in, I'm getting back 2544 saved albums with duplicates. I'm also using the semaphore pattern to limit the number of goroutines so that I don't exceed the rate limit.
I assume that the issue is with using the active variable rather than channels, but my attempt at that just resulted in an infinite loop
wg := &sync.WaitGroup{}
sem := make(chan bool, 20)
active := true
offset := 0
for {
sem <- true
if active {
// add each new goroutine to waitgroup
wg.Add(1)
go func() error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
// release the worker
defer func() { <-sem }()
savedAlbums, err := client.CurrentUsersAlbums(ctx, spotify.Limit(50), spotify.Offset(offset))
if err != nil {
return err
}
userAlbums = append(userAlbums, savedAlbums.Albums...)
if len(savedAlbums.Albums) < 50 {
// since the limit is set to 50, we know that if the number of returned albums
// is less than 50 that we're done retrieving data
active = false
return nil
} else {
offset += 50
return nil
}
}()
} else {
wg.Wait()
break
}
}
Thanks in advance!
I suspect that your main issue may be a misunderstanding of what the go keyword does; from the docs:
A "go" statement starts the execution of a function call as an independent concurrent thread of control, or goroutine, within the same address space.
So go func() error { starts the execution of the closure; it does not mean that any of the code runs immediately. In fact because, client.CurrentUsersAlbums will take a while, it's likely you will be requesting the first 50 items 20 times. This can be demonstrated with a simplified version of your application (playground)
func main() {
wg := &sync.WaitGroup{}
sem := make(chan bool, 20)
active := true
offset := 0
for {
sem <- true
if active {
// add each new goroutine to waitgroup
wg.Add(1)
go func() error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
// release the worker
defer func() { <-sem }()
fmt.Println("Getting from:", offset)
time.Sleep(time.Millisecond) // Simulate the query
// Pretend that we got back 50 albums
offset += 50
if offset > 2000 {
active = false
}
return nil
}()
} else {
wg.Wait()
break
}
}
}
Running this will produce somewhat unpredictable results (note that the playground caches results so try it on your machine) but you will probably see 20 X Getting from: 0.
A further issue is data races. Updating a variable from multiple goroutines without protection (e.g. sync.Mutex) results in undefined behaviour.
You will want to know how to fix this but unfortunately you will need to rethink your algorithm. Currently the process you are following is:
Set pos to 0
Get 50 records starting from pos
If we got 50 records then pos=pos+50 and loop back to step 2
This is a sequential algorithm; you don't know whether you have all of the data until you have requested the previous section. I guess you could make speculative queries (and handle failures) but a better solution would be to find some way to determine the number of results expected and then split the queries to get that number of records between multiple goroutines.
Note that if you do know the number of responses then you can do something like the following (playground):
noOfResultsToGet := 1644 // In the below we are getting 0-1643
noOfResultsPerRequest := 50
noOfSimultaneousRequests := 20 // You may not need this but many services will limit the number of simultaneous requests you can make (or, at least, rate limit them)
requestChan := make(chan int) // Will be passed the starting #
responseChan := make(chan []string) // Response from whatever request we are making (can be any type really)
// Start goroutines to make the requests
var wg sync.WaitGroup
wg.Add(noOfSimultaneousRequests)
for i := 0; i < noOfSimultaneousRequests; i++ {
go func(routineNo int) {
defer wg.Done()
for startPos := range requestChan {
// Simulate making the request
maxResult := startPos + noOfResultsPerRequest
if maxResult > noOfResultsToGet {
maxResult = noOfResultsToGet
}
rsp := make([]string, 0, noOfResultsPerRequest)
for x := startPos; x < maxResult; x++ {
rsp = append(rsp, strconv.Itoa(x))
}
responseChan <- rsp
fmt.Printf("Goroutine %d handling data from %d to %d\n", routineNo, startPos, startPos+noOfResultsPerRequest)
}
}(i)
}
// Close the response channel when all goroutines have shut down
go func() {
wg.Wait()
close(responseChan)
}()
// Send the requests
go func() {
for reqFrom := 0; reqFrom < noOfResultsToGet; reqFrom += noOfResultsPerRequest {
requestChan <- reqFrom
}
close(requestChan) // Allow goroutines to exit
}()
// Receive responses (note that these may be out of order)
result := make([]string, 0, noOfResultsToGet)
for x := range responseChan {
result = append(result, x...)
}
// Order the results and output (results from gorouting may come back in any order)
sort.Slice(result, func(i, j int) bool {
a, _ := strconv.Atoi(result[i])
b, _ := strconv.Atoi(result[j])
return a < b
})
fmt.Printf("Result: %v", result)
Relying on channels to pass messages often makes this kind of thing easier to think about and reduces the chance that you will make a mistake.
Set offset as an args -> go func(offset int) error {.
Increment offset by 50 after calling go func
Change active type to chan bool
To avoid data race on userAlbums = append(userAlbums, res...). We need to create channel that same type as userAlbums, then run for loop inside goroutine, then send the results to that channel.
this is the example : https://go.dev/play/p/yzk8qCURZFC
if applied to your code :
wg := &sync.WaitGroup{}
worker := 20
active := make(chan bool, worker)
for i := 0; i < worker; i++ {
active <- true
}
// I assume the type of userAlbums is []string
resultsChan := make(chan []string, worker)
go func() {
offset := 0
for {
if <-active {
// add each new goroutine to waitgroup
wg.Add(1)
go func(offset int) error {
// remove from waitgroup when goroutine is complete
defer wg.Done()
savedAlbums, err := client.CurrentUsersAlbums(ctx, spotify.Limit(50), spotify.Offset(offset))
if err != nil {
// active <- false // maybe you need this
return err
}
resultsChan <- savedAlbums.Albums
if len(savedAlbums.Albums) < 50 {
// since the limit is set to 50, we know that if the number of returned albums
// is less than 50 that we're done retrieving data
active <- false
return nil
} else {
active <- true
return nil
}
}(offset)
offset += 50
} else {
wg.Wait()
close(resultsChan)
break
}
}
}()
for res := range resultsChan {
userAlbums = append(userAlbums, res...)
}
I have the following piece of code. I'm trying to run 3 GO routines at the same time never exceeding three. This works as expected, but the code is supposed to be running updates a table in the DB.
So the first routine processes the first 50, then the second 50, and then third 50, and it repeats. I don't want two routines processing the same rows at the same time and due to how long the update takes, this happens almost every time.
To solve this, I started flagging the rows with a new column processing which is a bool. I set it to true for all rows to be updated when the routine starts and sleep the script for 6 seconds to allow the flag to be updated.
This works for a random amount of time, but every now and then, I'll see 2-3 jobs processing the same rows again. I feel like the method I'm using to prevent duplicate updates is a bit janky and was wondering if there was a better way.
stopper := make(chan struct{}, 3)
var counter int
for {
counter++
stopper <- struct{}{}
go func(db *sqlx.DB, c int) {
fmt.Println("start")
updateTables(db)
fmt.Println("stop"b)
<-stopper
}(db, counter)
time.Sleep(6 * time.Second)
}
in updateTables
var ids[]string
err := sqlx.Select(db, &data, `select * from table_data where processing = false `)
if err != nil {
panic(err)
}
for _, row:= range data{
list = append(ids, row.Id)
}
if len(rows) == 0 {
return
}
for _, row:= range data{
_, err = db.Exec(`update table_data set processing = true where id = $1, row.Id)
if err != nil {
panic(err)
}
}
// Additional row processing
I think there's a misunderstanding on approach to go routines in this case.
Go routines to do these kind of work should be approached like worker Threads, using channels as the communication method in between the main routine (which will be doing the synchronization) and the worker go routines (which will be doing the actual job).
package main
import (
"log"
"sync"
"time"
)
type record struct {
id int
}
func main() {
const WORKER_COUNT = 10
recordschan := make(chan record)
var wg sync.WaitGroup
for k := 0; k < WORKER_COUNT; k++ {
wg.Add(1)
// Create the worker which will be doing the updates
go func(workerID int) {
defer wg.Done() // Marking the worker as done
for record := range recordschan {
updateRecord(record)
log.Printf("req %d processed by worker %d", record.id, workerID)
}
}(k)
}
// Feeding the records channel
for _, record := range fetchRecords() {
recordschan <- record
}
// Closing our channel as we're not using it anymore
close(recordschan)
// Waiting for all the go routines to finish
wg.Wait()
log.Println("we're done!")
}
func fetchRecords() []record {
result := []record{}
for k := 0; k < 100; k++ {
result = append(result, record{k})
}
return result
}
func updateRecord(req record) {
time.Sleep(200 * time.Millisecond)
}
You can even buffer things in the main go routine if you need to update all the 50 tables at once.
I have a channel:
aChan := make(chan struct{})
and a timeout duration var t time.Duration. I want the program to exit either if the channel is closed, or the t timeout is reached,
if t is a positive duration.
I know I can use an outer if else loop, but this looks very redundant:
if t >= time.Duration(0) {
select {
case <-time.After(t):
fmt.Fprintln(os.Stdout, "timeout!"))
close(timeoutChan)
case <-aChan:
fmt.Fprintln(os.Stdout, "aChan is closed"))
return
}
} else {
select {
case <-aChan:
fmt.Fprintln(os.Stdout, "aChan is closed"))
return
}
}
Is there a more elegant way to write this?
Use a nil channel for the timeout when the duration is less than zero. The timeout case with a nil channel is not executed because receive on a nil channel is never ready.
var after <-chan time.Time
if t >= 0 {
after = time.After(t)
}
select {
case <-after:
fmt.Println("timeout!")
close(timeoutChan)
case <-aChan:
fmt.Println("aChan is closed")
return
}
I'm writing telegram-bot using Go (go-telegram-bot-api). It constantly parses a website and notifies about changes (every minute). Today I added simple keyboard and now can't make them work together.
The problem is that webpage parsing is in the infinite loop and program when it enters the loop ignores updates coming from the "client".
I've tried putting everything in a single loop, changing orders, etc. Maybe there is another or proper way of doing this?
Sample code:
u := tgbotapi.NewUpdate(0)
u.Timeout = 60
updates, err := bot.GetUpdatesChan(u)
for update := range updates {
if update.Message != nil {
switch update.Message.Text {
case "Show Keyboard":
Keyboard is sent
case "OptionsForParsing":
options.applied
case "StartParsing":
search bool = true
case "StopParsing":
search bool = false
}
}
if search == true{
for{
time.Sleep(1 * time.Minute)
AreThereChanges?()
if yes{
msg := tgbotapi.NewMessage(update.Message.Chat.ID, "help please")
bot.Send(msg)
}
}}
}
If I understood everything right you should start parsing loop inside of separated goroutine so updates cycle will keep working.
Try to do something like this:
// you can use cancel to stop running parsing goroutine
ctx, cancel := context.WithCancel(context.Background())
go func() {
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
AreThereChanges?()
if yes {
msg := tgbotapi.NewMessage(update.Message.Chat.ID, "help please")
bot.Send(msg)
}
}
}
}()
Writing from the top of my brain, so do not expect it to compile from the first attempt, but demonstrates the idea. Please, check out following links for more information regarding Ticker and contexts with cancellation: click, click.
Thanks to kimgrey I dug a bit into goroutines and channels. Infinite loop is in a separate goroutine and escaping is done through unbuffered channel (check "game changer" comments in the sample code). Now I can enter and escape infinite loop when needed. Here's how I solved my problem:
u := tgbotapi.NewUpdate(0)
u.Timeout = 60
updates, err := bot.GetUpdatesChan(u)
done := make(chan bool) //game changer
for update := range updates {
if update.Message != nil {
switch update.Message.Text {
case "Show Keyboard":
Keyboard is sent
case "OptionsForParsing":
options.applied
case "StartParsing":
search bool = true
case "StopParsing":
search bool = false
done<-true //game changer
}
go func() {
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
if search{
for {
select {
case <-done: //game changer
return
case <-ticker.C:
AnyChanges?()
}
}}
}()
}
}
I'm reading values from a channel in a loop like this:
for {
capturedFrame := <-capturedFrameChan
remoteCopy(capturedFrame)
}
To make it more efficient, I would like to read these values in a batch, with something like this (pseudo-code):
for {
capturedFrames := <-capturedFrameChan
multipleRemoteCopy(capturedFrames)
}
But I'm not sure how to do that. If I call capturedFrames := <-capturedFrameChan multiple times it's going to block.
Basically, what I would like is to read all the available values in captureFrameChan and, if none is available, it blocks as usual.
What would be the way to accomplish this in Go?
Something like this should work:
for {
// we initialize our slice. You may want to add a larger cap to avoid multiple memory allocations on `append`
capturedFrames := make([]Frame, 1)
// We block waiting for a first frame
capturedFrames[0] = <-capturedFrameChan
forLoop:
for {
select {
case buf := <-capturedFrameChan:
// if there is more frame immediately available, we add them to our slice
capturedFrames = append(capturedFrames, buf)
default:
// else we move on without blocking
break forLoop
}
}
multipleRemoteCopy(capturedFrames)
}
Try this (for channel ch with type T):
for firstItem := range ch { // For ensure that any batch could not be empty
var itemsBatch []T
itemsBatch = append(itemsBatch, firstItem)
Remaining:
for len(itemsBatch) < BATCHSIZE { // For control maximum size of batch
select {
case item := <-ch:
itemsBatch = append(itemsBatch, item)
default:
break Remaining
}
}
// Consume itemsBatch here...
}
But, if BATCHSIZE is constant, this code would be more efficient:
var i int
itemsBatch := [BATCHSIZE]T{}
for firstItem := range ch { // For ensure that any batch could not be empty
itemsBatch[0] = firstItem
Remaining:
for i = 1; i < BATCHSIZE; i++ { // For control maximum size of batch
select {
case itemsBatch[i] = <-ch:
default:
break Remaining
}
}
// Now you have itemsBatch with length i <= BATCHSIZE;
// Consume that here...
}
By using len(capturedFrames), you can do it like below:
for {
select {
case frame := <-capturedFrames:
frames := []Frame{frame}
for i := 0; i < len(capturedFrames); i++ {
frames = append(frames, <-capturedFrames)
}
multipleRemoteCopy(frames)
}
}
Seems you can also benchmark just
for {
capturedFrame := <-capturedFrameChan
go remoteCopy(capturedFrame)
}
without any codebase refactoring to see if it increase efficiency.
I've ended up doing it as below. Basically I've used len(capturedFrames) to know how many frames are available, then retrieved them in a loop:
for {
var paths []string
itemCount := len(capturedFrames)
if itemCount <= 0 {
time.Sleep(50 * time.Millisecond)
continue
}
for i := 0; i < itemCount; i++ {
f := <-capturedFrames
paths = append(paths, f)
}
err := multipleRemoteCopy(paths, opts)
if err != nil {
fmt.Printf("Error: could not remote copy \"%s\": %s", paths, err)
}
}