I have a time ticker that will execute a function within time interval(eg every 5 minutes, 10 minutes). I create this time ticker within a goroutine. I heard that this kind of ticker could leak the memory even if the app stopped. This ticker will keep running as long as the app running. Should it stop? how to stop it properly? Here is my implementation:
go func() {
for range time.Tick(5 * time.Minute) {
ExecuteFunctionA()
}
}()
What is the proper implementation for time ticker like this?
You can use a channel as a intermediary to stop the ticker properly.
Usually, I did it as the following:
var stopChan chan bool = make(chan bool)
func Stop() {
stopChan <- true
}
func Run() {
go func() {
ticker := time.NewTicker(5 * time.Minute)
for {
select {
case <- c.stopChan:
ticker.Stop()
return
case <- ticker.C:
ExecuteFunctionA()
}
}
}()
}
And you can invoke Stop function at the time you want to stop the ticker safely.
Nothing will leak if the 'app stopped'. The warning in the documentation refers to the fact that the garbage collector will not be able to reclaim the channel once it is created (time.Tick() initializes and returns a chan) and it will sit in memory even if you decide to break out of your for loop.
Based on your description in the question, this shouldn't be an issue for you since you want the ticker running as long as the app is running. But if you decide otherwise, you can use an alternative way like:
go func() {
for {
time.Sleep(time.Duration(5) * time.Minute)
go ExecuteFunctionA()
if someConditionIsMet {
break // nothing leaks in this case
}
}
}()
Related
I have written an API that makes DB calls and does some business logic. I am invoking a goroutine that must perform some operation in the background.
Since the API call should not wait for this background task to finish, I am returning 200 OK immediately after calling the goroutine (let us assume the background task will never give any error.)
I read that goroutine will be terminated once the goroutine has completed its task.
Is this fire and forget way safe from a goroutine leak?
Are goroutines terminated and cleaned up once they perform the job?
func DefaultHandler(w http.ResponseWriter, r *http.Request) {
// Some DB calls
// Some business logics
go func() {
// some Task taking 5 sec
}()
w.WriteHeader(http.StatusOK)
}
I would recommend always having your goroutines under control to avoid memory and system exhaustion.
If you are receiving a spike of requests and you start spawning goroutines without control, probably the system will go down soon or later.
In those cases where you need to return an immediate 200Ok the best approach is to create a message queue, so the server only needs to create a job in the queue and return the ok and forget. The rest will be handled by a consumer asynchronously.
Producer (HTTP server) >>> Queue >>> Consumer
Normally, the queue is an external resource (RabbitMQ, AWS SQS...) but for teaching purposes, you can achieve the same effect using a channel as a message queue.
In the example you'll see how we create a channel to communicate 2 processes.
Then we start the worker process that will read from the channel and later the server with a handler that will write to the channel.
Try to play with the buffer size and job time while sending curl requests.
package main
import (
"fmt"
"log"
"net/http"
"time"
)
/*
$ go run .
curl "http://localhost:8080?user_id=1"
curl "http://localhost:8080?user_id=2"
curl "http://localhost:8080?user_id=3"
curl "http://localhost:8080?user_id=....."
*/
func main() {
queueSize := 10
// This is our queue, a channel to communicate processes. Queue size is the number of items that can be stored in the channel
myJobQueue := make(chan string, queueSize) // Search for 'buffered channels'
// Starts a worker that will read continuously from our queue
go myBackgroundWorker(myJobQueue)
// We start our server with a handler that is receiving the queue to write to it
if err := http.ListenAndServe("localhost:8080", myAsyncHandler(myJobQueue)); err != nil {
panic(err)
}
}
func myAsyncHandler(myJobQueue chan<- string) http.HandlerFunc {
return func(rw http.ResponseWriter, r *http.Request) {
// We check that in the query string we have a 'user_id' query param
if userID := r.URL.Query().Get("user_id"); userID != "" {
select {
case myJobQueue <- userID: // We try to put the item into the queue ...
rw.WriteHeader(http.StatusOK)
rw.Write([]byte(fmt.Sprintf("queuing user process: %s", userID)))
default: // If we cannot write to the queue it's because is full!
rw.WriteHeader(http.StatusInternalServerError)
rw.Write([]byte(`our internal queue is full, try it later`))
}
return
}
rw.WriteHeader(http.StatusBadRequest)
rw.Write([]byte(`missing 'user_id' in query params`))
}
}
func myBackgroundWorker(myJobQueue <-chan string) {
const (
jobDuration = 10 * time.Second // simulation of a heavy background process
)
// We continuosly read from our queue and process the queue 1 by 1.
// In this loop we could spawn more goroutines in a controlled way to paralelize work and increase the read throughput, but i don't want to overcomplicate the example.
for userID := range myJobQueue {
// rate limiter here ...
// go func(u string){
log.Printf("processing user: %s, started", userID)
time.Sleep(jobDuration)
log.Printf("processing user: %s, finisehd", userID)
// }(userID)
}
}
There is no "goroutine cleaning" you have to handle, you just launch goroutines and they'll be cleaned when the function launched as a goroutine returns. Quoting from Spec: Go statements:
When the function terminates, its goroutine also terminates. If the function has any return values, they are discarded when the function completes.
So what you do is fine. Note however that your launched goroutine cannot use or assume anything about the request (r) and response writer (w), you may only use them before you return from the handler.
Also note that you don't have to write http.StatusOK, if you return from the handler without writing anything, that's assumed to be a success and HTTP 200 OK will be sent back automatically.
See related / possible duplicate: Webhook process run on another goroutine
#icza is absolutely right there is no "goroutine cleaning" you can use a webhook or a background job like gocraft. The only way I can think of using your solution is to use the sync package for learning purposes.
func DefaultHandler(w http.ResponseWriter, r *http.Request) {
// Some DB calls
// Some business logics
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
// some Task taking 5 sec
}()
w.WriteHeader(http.StatusOK)
wg.wait()
}
you can wait for a goroutine to finish using &sync.WaitGroup:
// BusyTask
func BusyTask(t interface{}) error {
var wg = &sync.WaitGroup{}
wg.Add(1)
go func() {
// busy doing stuff
time.Sleep(5 * time.Second)
wg.Done()
}()
wg.Wait() // wait for goroutine
return nil
}
// this will wait 5 second till goroutune finish
func main() {
fmt.Println("hello")
BusyTask("some task...")
fmt.Println("done")
}
Other way is to attach a context.Context to goroutine and time it out.
//
func BusyTaskContext(ctx context.Context, t string) error {
done := make(chan struct{}, 1)
//
go func() {
// time sleep 5 second
time.Sleep(5 * time.Second)
// do tasks and signle done
done <- struct{}{}
close(done)
}()
//
select {
case <-ctx.Done():
return errors.New("timeout")
case <-done:
return nil
}
}
//
func main() {
fmt.Println("hello")
ctx, cancel := context.WithTimeout(context.TODO(), 2*time.Second)
defer cancel()
if err := BusyTaskContext(ctx, "some task..."); err != nil {
fmt.Println(err)
return
}
fmt.Println("done")
}
I have a handler function for an endpoint. The handler takes a very long time to return a response, consists a lot of processing. I do not want other incoming requests to run concurrently but instead wait for the previous one to finish! Tried implementing waitGroups, check the code! Every time for a new request a new instance of wait group is created and it starts running concurrently instead of waiting for the older one to complete. Is my wait group approach incorrect?
var wg sync.WaitGroup
func Handler(c *gin.Context) {
// some stuff that takes ~10-15 seconds, can't be run concurrently
// If a second request comes put it in a queue and execute it only once this is done
wg.Add(1)
go func() {
defer wg.Done()
//some processing happens
time.Sleep(10 * time.Second)
}()
wg.Wait()
c.JSON(http.StatusOK, gin.H{"message": "Hello!"})
}
router.POST("/doSomething", Handler)
As already mentioned in the comments, this looks like a broken requirement. However if you really want to have one instance of the function running, you can use a mutex:
var lock sync.Mutex
func Handler(c *gin.Context) {
lock.Lock()
defer lock.Unlock()
// Process
}
I'm new to go and trying to make two methods run at the same time periodically for as long at the application is running. I've managed to come up with the following but the for true part does not feel right as this is blocking.
Would channels be a better way todo this? Any pointers in the right direction would be helpful.
func main() {
t1 := schedule(ping, time.Second)
t2 := schedule(ping, 2*time.Second)
for true {
time.Sleep(1 * time.Second)
}
t1.Stop()
t2.Stop()
}
func schedule(f func(interval time.Duration), interval time.Duration) *time.Ticker {
ticker := time.NewTicker(interval)
go func() {
for range ticker.C {
f(interval)
}
}()
return ticker
}
func ping(interval time.Duration) {
log.Println("ping ", interval)
}
To prevent the application from exiting, the main goroutine must block.
Use select {} to block the main goroutine.
Because the tickers run for the duration of the application, there's no need to stop the tickers.
func main() {
schedule(ping, time.Second)
schedule(ping, 2*time.Second)
select {}
}
Is there any native library or third party support like ScheduledExecutorService by java native library at go lang for production use case?
Please find the code snippet in java 1.8 :
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class TaskScheduler {
/**
* #param args
*/
public static void main(String[] args) {
Runnable runnable = ()-> {
// task to run goes here
System.out.println("Hello !!");
};
ScheduledExecutorService service = Executors.newScheduledThreadPool(1);
service.scheduleAtFixedRate(runnable, 0, 1, TimeUnit.SECONDS);
}
}
It will print Hello !! in every one second.
No need to use 3rd party library to achieve that. Simply take the advantage of goroutine and use available time.Sleep() API from time package, then the very same result can be achieved.
Example:
go func() {
for true {
fmt.Println("Hello !!")
time.Sleep(1 * time.Second)
}
}()
Playground: https://play.golang.org/p/IMV_IAt-VQX
Example using ticker #1
As per suggestion from Siddhanta. Here is one example to achieve the same result by using ticker (taken from go documentation page of ticker, with some modifications following your requirement).
done := make(chan bool)
ticker := time.NewTicker(1 * time.Second)
go func() {
for {
select {
case <-done:
ticker.Stop()
return
case <-ticker.C:
fmt.Println("Hello !!")
}
}
}()
// wait for 10 seconds
time.Sleep(10 *time.Second)
done <- true
The ticker time information (the time when the Hello !! executed) can be taken from ticker.C channel.
case t := <-ticker.C:
fmt.Println(t)
Playground: https://play.golang.org/p/TN2M-AMr39L
Example using ticker #2
Another simplified example of ticker, taken from https://gobyexample.com/tickers
ticker := time.NewTicker(1 * time.Second)
go func() {
for range ticker.C {
fmt.Println("Hello !!")
}
}()
// wait for 10 seconds
time.Sleep(10 *time.Second)
ticker.Stop()
I have written some code that will concurrently poll URLs every 30 minutes:
func (obj * MyObj) Poll() {
for ;; {
for _, url := range obj.UrlList {
//Download the current contents of the URL and do something with it
}
time.Sleep(30 * time.Minute)
}
//Start the routine in another function
go obj.Poll()
How would I then add to obj.UrlList elsewhere in the code and ensure that the next time the URLs are polled that the UrlList in the Poll goroutine as also been updated and as such will also poll the new URL?
I understand that memory is shared through communicating rather than vice versa in Go and I've investigated channels however I'm not sure how to implement them in this example.
Here's an untested, but safe model for periodically fetching some URLs with the ability to dynamically add new URLs to the list of URLs safely. It should be obvious to the reader what would be required if you wanted to remove a URL as well.
type harvester struct {
ticker *time.Ticker // periodic ticker
add chan string // new URL channel
urls []string // current URLs
}
func newHarvester() *harvester {
rv := &harvester{
ticker: time.NewTicker(time.Minute * 30),
add: make(chan string),
}
go rv.run()
return rv
}
func (h *harvester) run() {
for {
select {
case <-h.ticker.C:
// When the ticker fires, it's time to harvest
for _, u := range h.urls {
harvest(u)
}
case u := <-h.add:
// At any time (other than when we're harvesting),
// we can process a request to add a new URL
h.urls = append(h.urls, u)
}
}
}
func (h *harvester) AddURL(u string) {
// Adding a new URL is as simple as tossing it onto a channel.
h.add <- u
}
If you need to poll at regular periodic intervals, you should not use time.Sleep but a time.Ticker instead (or relative like time.After). The reason is that a sleep is just a sleep and takes no account of drift due to the real work you did in your loop. Conversely, a Ticker has a separate goroutine and a channel, which together are able to send you regular events and thereby cause something useful to happen.
Here's an example that's similar to yours. I put in a random jitter to illustrate the benefit of using a Ticker.
package main
import (
"fmt"
"time"
"math/rand"
)
func Poll() {
r := rand.New(rand.NewSource(99))
c := time.Tick(10 * time.Second)
for _ = range c {
//Download the current contents of the URL and do something with it
fmt.Printf("Grab at %s\n", time.Now())
// add a bit of jitter
jitter := time.Duration(r.Int31n(5000)) * time.Millisecond
time.Sleep(jitter)
}
}
func main() {
//go obj.Poll()
Poll()
}
When I ran this, I found that it kept to a strict 10-second cycle in spite of the jitter.
// Type with queue through a channel.
type MyType struct {
queue chan []*net.URL
}
func (t *MyType) poll() {
for urls := range t.queue {
...
time.Sleep(30 * time.Minute)
}
}
// Create instance with buffered queue.
t := MyType{make(chan []*net.URL, 25)}
go t.Poll()