Go routine leak fix - go

I am working on a small service at the moment. From my testing, the code I've written has the possibility of leaking go routines under certain circumstances pertaining to the context. Is there a good and/or idiomatic way to remedy this? I'm providing some sample code below.
func Handle(ctx context.Context, r *Req) (*Response, error) {
ctx, cancel := context.WithTimeout(ctx, time.Second * 5)
defer cancel()
resChan := make(chan Response)
errChan := make(chan error)
go process(r, resChan, errChan)
select {
case ctx.Done():
return nil, ctx.Err()
case res := <-resChan:
return &res, nil
case err := <-errChan:
return nil, err
}
}
func process(r *Req, resChan chan<- Response, errChan chan<- error) {
defer close(errChan)
defer close(resChan)
err := doSomeWork()
if err != nil {
errChan <- err
return
}
err = doSomeMoreWork()
if err != nil {
errChan <- err
return
}
res := Response{}
resChan <- res
}
Hypothetically, if the client cancelled the context or the timeout occurred before the process func had a chance to send on one of the unbuffered channels (resChan, errChan), there would be no channel readers left from Handle and sending on the channels would block indefinitely with no readers. Since process would not return in this case, the channels would also not be closed.
I came up with the process2 as a solution, but I can't help thinking I'm doing something wrong, or there's a better way to handle this.
func process2(ctx context.Context, r *Req, resChan chan<- Response, errChan chan<- error) {
defer close(errChan)
defer close(resChan)
err := doSomeWork()
select {
case <-ctx.Done():
return
default:
if err != nil {
errChan <- err
return
}
}
err = doSomeMoreWork()
select {
case <-ctx.Done():
return
default:
if err != nil {
errChan <- err
return
}
}
res := Response{}
select{
case <-ctx.Done():
return
default:
resChan <- res
}
}
This approach makes sure that each time a channel send is attempted, first the context is checked for having been completed or cancelled. If it was, then it does not attempt the send and returns. I'm pretty sure this fixes any go routine leaking happening in the first process func.
Is there a better way? Maybe I have this all wrong.

Related

websocket gracefully shutdown

I have a websocket server. I wrote a test for him that tests his ability to shutdown gracefully. 5 connections are created and each sends 5 requests. After a while, shutdown starts. All 25 requests must be fulfilled. If I close the exit channel, then the test does not work as it should.
time.AfterFunc(50*time.Millisecond, func() {
close(exit)
close(done)
})
And if I just call the s.shutdown function, then everything is ok.
time.AfterFunc(50*time.Millisecond, func() {
require.Nil(t, s.Shutdown())
close(done)
})
My test
func TestServer_GracefulShutdown(t *testing.T) {
done := make(chan struct{})
exit := make(chan struct{})
ctx := context.Background()
finishedRequestCount := atomic.NewInt32(0)
ln, err := net.Listen("tcp", "localhost:")
require.Nil(t, err)
handler := HandlerFunc(func(conn *websocket.Conn) {
for {
_, _, err := conn.ReadMessage()
if err != nil {
return
}
time.Sleep(100 * time.Millisecond)
finishedRequestCount.Inc()
}
})
s, err := makeServer(ctx, handler) // server create
require.Nil(t, err)
time.AfterFunc(50*time.Millisecond, func() {
close(exit)
close(done)
})
go func() {
fmt.Printf("Running...")
require.Nil(t, s.Run(ctx, exit, ln))
}()
for i := 0; i < connCount; i++ {
go func() {
err := clientRun(ln)
require.Nil(t, err)
}()
}
<-done
assert.Equal(t, int32(totalCount), finishedRequestCount.Load())
}
My run func
func (s *Server) Run(ctx context.Context, exit <-chan struct{}, ln net.Listener) error {
errs := make(chan error, 1)
go func() {
err := s.httpServer.Run(ctx, exit, ln)
if err != nil {
errs <- err
}
}()
select {
case <-ctx.Done():
return s.Close()
case <-exit:
return s.Shutdown()
case err := <-errs:
return err
}
}
My shutdown
func (s *Server) Shutdown() error {
err := s.httpServer.Shutdown() // we close the possibility to connect to any conn
s.wg.Wait()
return err
}
What is happening if the following code is executed?
close(exit)
close(done)
Both channels are closed almost at the same time. The first triggers the Shutdown function which waits for a graceful shutdown. But the second triggers the evaluation of
assert.Equal(t, int32(totalCount), finishedRequestCount.Load())
It is triggered while the graceful shutdown is still running or hasn't even started yet.
If you execute the Shutdown function directly it will block until finished and only then close(done) will start the assertion. That is why this works:
require.Nil(t, s.Shutdown())
close(done)
You can move the close(done) to the following location to make the test work while using the exit channel to close:
go func() {
fmt.Printf("Running...")
require.Nil(t, s.Run(ctx, exit, ln))
close(done)
}()
This way done will be closed after the Shutdown function was executed.
As discussed in the comments I strongly suggest to use contexts instead of channels to close. They have the complexity of close channels hidden away.

Go routines with kafka consumer channel and context

I have a simple kafka consumer for which I have created a handle and trying to read it using a go routine:
func process(ctx context.Context){
consumer := queueHandle.Consume(topic_ops_req, consumerHandler)
// Get signal for finish
doneCh := make(chan struct{})
go func(consumer chan *sarama.ConsumerMessage, ctx context.Context) {
for {
select {
case msg, ok := <-consumer:
if !ok {
logger.Info("Channel has been closed")
doneCh <- struct{}{}
return
}
var request queue.Request
err := json.Unmarshal(msg.Value, &request)
if err != nil {
logger.Error("consumer unmarshal err", err)
panic(err)
}
res, err := new_process(ctx, request, service) // call another func
if err != nil {
//TODO
}
result = res
doneCh <- struct{}{}
case <-ctx.Done():
logger.Info(fmt.Sprintf("Context ended with err : %s", ctx.Err()))
doneCh <- struct{}{}
}
}
}(consumer, ctx)
<-doneCh
}
The issue I am seeing is that once I introduce the "case <-ctx.Done()", the go routine does not enter the "case msg, ok := <-consumer" and always returns that the context ended. How do I my go func work with both consumer channel and ctx.Done() ?

how to return values in a goroutine

I have the code:
go s.addItemSync(ch, cs.ResponseQueue, user)
This calls the func:
func (s *Services) addItemSync(ch types.ChannelInsertion, statusQueueName, user string) {
//func body here
}
I would however like to do this:
if ok, err := go s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
return
}
Which would change the other func to this
func (s *Services) addItemSync(ch types.ChannelInsertion, statusQueueName, user string) (bool, error) {
}
As in, I would like to be able to declare a go func but this errors out every time. Any idea how you can declare a variable while able to call the go func ability for synchronous calls? as seen in the if ok, err := go s.addItemSync(ch, cs.ResponseQueue, user); !ok { line?
If you want to wait until a go-routine has completed, you need to return results in a channel. The basic pattern, without complicating with wait groups, etc. is:
func myFunc() {
// make a channel to receive errors
errChan := make(chan error)
// launch a go routine
go doSomething(myVar, errChan)
// block until something received on the error channel
if err := <- errChan; err != nil {
// something bad happened
}
}
// your async funciton
func doSomething(myVar interface{}, errChan chan error) {
// Do stuff
if something, err := someOtherFunc(myVar); err != nil {
errChan <- err
return
}
// all good - send nil to the error channel
errChan <- nil
}
In your case if you just want to fire off a go-routine and log if an error happens, you can use an anonymous function:
go func() {
if ok, err := s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
}
}
}()
Or if you want to wait for the result:
errChan := make(chan error)
go func() {
if ok, err := s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
errChan <- err
return
}
}
errChan <- nil
}()
// do some other stuff while we wait...
// block until go routine returns
if err := <- errChan; err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
}
Note:
Your code as written, may have unexpected results if it is possible that a response where ok == false would not return an error. If this is a concern, I would suggest creating and returning a new error for cases where !ok && err == nil

What happens to a value never consumed from a Buffered channel

I have the following code
func f() {
...
chan := make(chan error, 1)
go func() {
...
chan <- err
}()
err := other_method()
if err != nil {
log(err)
return
}
err <- chan
if err != nil {
log(err)
}
}
What will happen to the value written in the buffered channel if it is never ever read, because the func exited before reading from it? Is this a resource leak I need to care about?

Go channel infinite loop

I am trying to catch errors from a group of goroutines using a channel, but the channel enters an infinite loop, starts consuming CPU.
func UnzipFile(f *bytes.Buffer, location string) error {
zipReader, err := zip.NewReader(bytes.NewReader(f.Bytes()), int64(f.Len()))
if err != nil {
return err
}
if err := os.MkdirAll(location, os.ModePerm); err != nil {
return err
}
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, errorList)
fileWaitGroup := &sync.WaitGroup{}
for _, file := range zipReader.File {
fileWaitGroup.Add(1)
go writeZipFileToLocal(file, location, errorChannel, fileWaitGroup)
}
fileWaitGroup.Wait()
close(errorChannel)
log.Println(errorList)
return nil
}
func errorChannelWatch(ch chan error, list []error) {
for {
select {
case err := <- ch:
list = append(list, err)
}
}
}
func writeZipFileToLocal(file *zip.File, location string, ch chan error, wg *sync.WaitGroup) {
defer wg.Done()
zipFilehandle, err := file.Open()
if err != nil {
ch <- err
return
}
defer zipFilehandle.Close()
if file.FileInfo().IsDir() {
if err := os.MkdirAll(filepath.Join(location, file.Name), os.ModePerm); err != nil {
ch <- err
}
return
}
localFileHandle, err := os.OpenFile(filepath.Join(location, file.Name), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, file.Mode())
if err != nil {
ch <- err
return
}
defer localFileHandle.Close()
if _, err := io.Copy(localFileHandle, zipFilehandle); err != nil {
ch <- err
return
}
ch <- fmt.Errorf("Test error")
}
So I am looping a slice of files and writing them to my disk, when there is an error I report back to the errorChannel to save that error into a slice.
I use a sync.WaitGroup to wait for all goroutines and when they are done I want to print errorList and check if there was any error during the execution.
The list is always empty, even if I add ch <- fmt.Errorf("test") at the end of writeZipFileToLocal and the channel always hangs up.
I am not sure what I am missing here.
1. For the first point, the infinite loop:
Citing from golang language spec:
A receive operation on a closed channel can always proceed
immediately, yielding the element type's zero value after any
previously sent values have been received.
So in this function
func errorChannelWatch(ch chan error, list []error) {
for {
select {
case err := <- ch:
list = append(list, err)
}
}
}
after ch gets closed this turns into an infinite loop adding nil values to list.
Try this instead:
func errorChannelWatch(ch chan error, list []error) {
for err := range ch {
list = append(list, err)
}
}
2. For the second point, why you don't see anything in your error list:
The problem is this call:
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, errorList)
Here you hand errorChannelWatch the errorList as a value. So the slice errorList will not be changed by the function. What is changed, is the underlying array, as long as the append calls don't need to allocate a new one.
To remedy the situation, either hand a slice pointer to errorChannelWatch or rewrite it as a call to a closure, capturing
errorList.
For the first proposed solution, change errorChannelWatch to
func errorChannelWatch(ch chan error, list *[]error) {
for err := range ch {
*list = append(*list, err)
}
}
and the call to
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, &errorList)
For the second proposed solution, just change the call to
errorChannel := make(chan error)
errorList := []error{}
go func() {
for err := range errorChannel {
errorList = append(errorList, err)
}
} ()
3. A minor remark:
One could think, that there is a synchronisation problem here:
fileWaitGroup.Wait()
close(errorChannel)
log.Println(errorList)
How can you be sure, that errorList isn't modified, after the call to close? One could reason, that you can't know, how many values the goroutine errorChannelWatch still has to process.
Your synchronisation seems correct to me, as you do the wg.Done()
after the send to the error channel and so all error values will
be sent, when fileWaitGroup.Wait() returns.
But that can change, if someone later adds a buffering to the error
channel or alters the code.
So I would advise to at least explain the synchronisation in a comment.

Resources