I have the code:
go s.addItemSync(ch, cs.ResponseQueue, user)
This calls the func:
func (s *Services) addItemSync(ch types.ChannelInsertion, statusQueueName, user string) {
//func body here
}
I would however like to do this:
if ok, err := go s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
return
}
Which would change the other func to this
func (s *Services) addItemSync(ch types.ChannelInsertion, statusQueueName, user string) (bool, error) {
}
As in, I would like to be able to declare a go func but this errors out every time. Any idea how you can declare a variable while able to call the go func ability for synchronous calls? as seen in the if ok, err := go s.addItemSync(ch, cs.ResponseQueue, user); !ok { line?
If you want to wait until a go-routine has completed, you need to return results in a channel. The basic pattern, without complicating with wait groups, etc. is:
func myFunc() {
// make a channel to receive errors
errChan := make(chan error)
// launch a go routine
go doSomething(myVar, errChan)
// block until something received on the error channel
if err := <- errChan; err != nil {
// something bad happened
}
}
// your async funciton
func doSomething(myVar interface{}, errChan chan error) {
// Do stuff
if something, err := someOtherFunc(myVar); err != nil {
errChan <- err
return
}
// all good - send nil to the error channel
errChan <- nil
}
In your case if you just want to fire off a go-routine and log if an error happens, you can use an anonymous function:
go func() {
if ok, err := s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
}
}
}()
Or if you want to wait for the result:
errChan := make(chan error)
go func() {
if ok, err := s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
errChan <- err
return
}
}
errChan <- nil
}()
// do some other stuff while we wait...
// block until go routine returns
if err := <- errChan; err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
}
Note:
Your code as written, may have unexpected results if it is possible that a response where ok == false would not return an error. If this is a concern, I would suggest creating and returning a new error for cases where !ok && err == nil
Related
I have two functions that I cannot change (see first() and second() below). They are returning some data and errors (the output data is different, but in the examples below I use (string, error) for simplicity)
I would like to run them in separate goroutines - my approach:
package main
import (
"fmt"
"os"
)
func first(name string) (string, error) {
if name == "" {
return "", fmt.Errorf("empty name is not allowed")
}
fmt.Println("processing first")
return fmt.Sprintf("First hello %s", name), nil
}
func second(name string) (string, error) {
if name == "" {
return "", fmt.Errorf("empty name is not allowed")
}
fmt.Println("processing second")
return fmt.Sprintf("Second hello %s", name), nil
}
func main() {
firstCh := make(chan string)
secondCh := make(chan string)
go func() {
defer close(firstCh)
res, err := first("one")
if err != nil {
fmt.Printf("Failed to run first: %v\n", err)
}
firstCh <- res
}()
go func() {
defer close(secondCh)
res, err := second("two")
if err != nil {
fmt.Printf("Failed to run second: %v\n", err)
}
secondCh <- res
}()
resultsOne := <-firstCh
resultsTwo := <-secondCh
// It's important for my app to do error checking and stop if errors exist.
if resultsOne == "" || resultsTwo == "" {
fmt.Println("There was an ERROR")
os.Exit(1)
}
fmt.Println("ONE:", resultsOne)
fmt.Println("TWO:", resultsTwo)
}
I believe one caveat is that resultsOne := <- firstCh blocks until first goroutine finishes, but I don't care too much about this.
Can you please confirm that my approach is good? What other approaches would be better in my situation?
The example looks mostly good. A couple improvements are:
declaring your channels as buffered
firstCh := make(chan string, 1)
secondCh := make(chan string, 1)
With unbuffered channels, send operations block (until someone receives). If your goroutine #2 is much faster than the first, it will have to wait until the first finishes as well, since you receive in sequence:
resultsOne := <-firstCh // waiting on this one first
resultsTwo := <-secondCh // sender blocked because the main thread hasn't reached this point
use "golang.org/x/sync/errgroup".Group. The program will feel "less native" but it dispenses you from managing channels by hand — which trades, in a non-contrived setting, for sync'ing writes on the results:
func main() {
var (
resultsOne string
resultsTwo string
)
g := errgroup.Group{}
g.Go(func() error {
res, err := first("one")
if err != nil {
return err
}
resultsOne = res
return nil
})
g.Go(func() error {
res, err := second("two")
if err != nil {
return err
}
resultsTwo = res
return nil
})
err := g.Wait()
// ... handle err
I am working on a small service at the moment. From my testing, the code I've written has the possibility of leaking go routines under certain circumstances pertaining to the context. Is there a good and/or idiomatic way to remedy this? I'm providing some sample code below.
func Handle(ctx context.Context, r *Req) (*Response, error) {
ctx, cancel := context.WithTimeout(ctx, time.Second * 5)
defer cancel()
resChan := make(chan Response)
errChan := make(chan error)
go process(r, resChan, errChan)
select {
case ctx.Done():
return nil, ctx.Err()
case res := <-resChan:
return &res, nil
case err := <-errChan:
return nil, err
}
}
func process(r *Req, resChan chan<- Response, errChan chan<- error) {
defer close(errChan)
defer close(resChan)
err := doSomeWork()
if err != nil {
errChan <- err
return
}
err = doSomeMoreWork()
if err != nil {
errChan <- err
return
}
res := Response{}
resChan <- res
}
Hypothetically, if the client cancelled the context or the timeout occurred before the process func had a chance to send on one of the unbuffered channels (resChan, errChan), there would be no channel readers left from Handle and sending on the channels would block indefinitely with no readers. Since process would not return in this case, the channels would also not be closed.
I came up with the process2 as a solution, but I can't help thinking I'm doing something wrong, or there's a better way to handle this.
func process2(ctx context.Context, r *Req, resChan chan<- Response, errChan chan<- error) {
defer close(errChan)
defer close(resChan)
err := doSomeWork()
select {
case <-ctx.Done():
return
default:
if err != nil {
errChan <- err
return
}
}
err = doSomeMoreWork()
select {
case <-ctx.Done():
return
default:
if err != nil {
errChan <- err
return
}
}
res := Response{}
select{
case <-ctx.Done():
return
default:
resChan <- res
}
}
This approach makes sure that each time a channel send is attempted, first the context is checked for having been completed or cancelled. If it was, then it does not attempt the send and returns. I'm pretty sure this fixes any go routine leaking happening in the first process func.
Is there a better way? Maybe I have this all wrong.
every one,I am new to golang.I wanna get the data from log file generated by my application.cuz roll-back mechanism, I met some problem.For instance,my target log file is chats.log,it will be renamed to chats.log.2018xxx and a new chats.log will be created.so my go routine that read log file will fail to work.
so I need detect the change and shutdown the previous go routine and then establish the new go routine.
I looked for modules that can help me,and I found
func ExampleNewWatcher(fn string, createnoti chan string, wg sync.WaitGroup) {
wg.Add(1)
defer wg.Done()
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
defer watcher.Close()
done := make(chan bool)
go func() {
for {
select {
case event := <-watcher.Events:
if event.Op == fsnotify.Create && event.Name==fn{
createnoti <- "has been created"
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}()
err = watcher.Add("./")
if err != nil {
log.Fatal(err)
}
<-done
}
I use fsnotify to detech the change,and make sure the event of file is my log file,and then send some message to a channel.
this is my worker go routine:
func tailer(fn string,isfollow bool, outchan chan string, done <-chan interface{},wg sync.WaitGroup) error {
wg.Add(1)
defer wg.Done()
_, err := os.Stat(fn)
if err != nil{
panic(err)
}
t, err := tail.TailFile(fn, tail.Config{Follow:isfollow})
if err != nil{
panic(err)
}
defer t.Stop()
for line := range t.Lines{
select{
case outchan <- line.Text:
case <- done:
return nil
}
}
return nil
}
I using tail module to read the log file,and I add a done channel to it to shutdown the cycle(I don't know whether I put it in the right way)
And I will send every log content to a channel to consuming it.
So here is the question:how should I put it together?
ps: Actually,I can use some tool to do this job.like apache-flume,but all of those tools need dependency.
Thank you a lot!
Here is a complete example that reloads and rereads the file as it changes or gets deleted and recreated:
package main
import (
"github.com/fsnotify/fsnotify"
"io/ioutil"
"log"
)
const filename = "myfile.txt"
func ReadFile(filename string) string {
data, err := ioutil.ReadFile(filename)
if err != nil {
log.Println(err)
}
return string(data)
}
func main() {
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatal(err)
}
defer watcher.Close()
err = watcher.Add("./")
if err != nil {
log.Fatal(err)
}
for {
select {
case event := <-watcher.Events:
if event.Op == fsnotify.Create && event.Name == filename {
log.Println(ReadFile(filename))
}
case err := <-watcher.Errors:
log.Println("error:", err)
}
}
}
Note this doesn't require goroutines, channels or a WaitGroup. Better to keep things simple and reserve those for when they're actually needed.
I am trying to catch errors from a group of goroutines using a channel, but the channel enters an infinite loop, starts consuming CPU.
func UnzipFile(f *bytes.Buffer, location string) error {
zipReader, err := zip.NewReader(bytes.NewReader(f.Bytes()), int64(f.Len()))
if err != nil {
return err
}
if err := os.MkdirAll(location, os.ModePerm); err != nil {
return err
}
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, errorList)
fileWaitGroup := &sync.WaitGroup{}
for _, file := range zipReader.File {
fileWaitGroup.Add(1)
go writeZipFileToLocal(file, location, errorChannel, fileWaitGroup)
}
fileWaitGroup.Wait()
close(errorChannel)
log.Println(errorList)
return nil
}
func errorChannelWatch(ch chan error, list []error) {
for {
select {
case err := <- ch:
list = append(list, err)
}
}
}
func writeZipFileToLocal(file *zip.File, location string, ch chan error, wg *sync.WaitGroup) {
defer wg.Done()
zipFilehandle, err := file.Open()
if err != nil {
ch <- err
return
}
defer zipFilehandle.Close()
if file.FileInfo().IsDir() {
if err := os.MkdirAll(filepath.Join(location, file.Name), os.ModePerm); err != nil {
ch <- err
}
return
}
localFileHandle, err := os.OpenFile(filepath.Join(location, file.Name), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, file.Mode())
if err != nil {
ch <- err
return
}
defer localFileHandle.Close()
if _, err := io.Copy(localFileHandle, zipFilehandle); err != nil {
ch <- err
return
}
ch <- fmt.Errorf("Test error")
}
So I am looping a slice of files and writing them to my disk, when there is an error I report back to the errorChannel to save that error into a slice.
I use a sync.WaitGroup to wait for all goroutines and when they are done I want to print errorList and check if there was any error during the execution.
The list is always empty, even if I add ch <- fmt.Errorf("test") at the end of writeZipFileToLocal and the channel always hangs up.
I am not sure what I am missing here.
1. For the first point, the infinite loop:
Citing from golang language spec:
A receive operation on a closed channel can always proceed
immediately, yielding the element type's zero value after any
previously sent values have been received.
So in this function
func errorChannelWatch(ch chan error, list []error) {
for {
select {
case err := <- ch:
list = append(list, err)
}
}
}
after ch gets closed this turns into an infinite loop adding nil values to list.
Try this instead:
func errorChannelWatch(ch chan error, list []error) {
for err := range ch {
list = append(list, err)
}
}
2. For the second point, why you don't see anything in your error list:
The problem is this call:
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, errorList)
Here you hand errorChannelWatch the errorList as a value. So the slice errorList will not be changed by the function. What is changed, is the underlying array, as long as the append calls don't need to allocate a new one.
To remedy the situation, either hand a slice pointer to errorChannelWatch or rewrite it as a call to a closure, capturing
errorList.
For the first proposed solution, change errorChannelWatch to
func errorChannelWatch(ch chan error, list *[]error) {
for err := range ch {
*list = append(*list, err)
}
}
and the call to
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, &errorList)
For the second proposed solution, just change the call to
errorChannel := make(chan error)
errorList := []error{}
go func() {
for err := range errorChannel {
errorList = append(errorList, err)
}
} ()
3. A minor remark:
One could think, that there is a synchronisation problem here:
fileWaitGroup.Wait()
close(errorChannel)
log.Println(errorList)
How can you be sure, that errorList isn't modified, after the call to close? One could reason, that you can't know, how many values the goroutine errorChannelWatch still has to process.
Your synchronisation seems correct to me, as you do the wg.Done()
after the send to the error channel and so all error values will
be sent, when fileWaitGroup.Wait() returns.
But that can change, if someone later adds a buffering to the error
channel or alters the code.
So I would advise to at least explain the synchronisation in a comment.
im looking for the most efficient way to invoke couple of method
together.
Basically what im trying to to is invoke those method together and if something went wrong return error else return the struct Type.
This code is working but i can't get the struct type or error and im not sure if its the correct way.
go func()(struct,err) {
struct,err= sm.MethodA()//return struct type or error
err = sm.MethodB()//return error or nill
return struct,err
}()
In Go, it's idiomatic to return the two values and check for nil against the error
For example:
func myFunc(sm SomeStruct) (MyStruct, error) {
s, err := sm.MethodA()
if err != nil {
return nil, err
}
if err := sm.MethodB(); err != nil {
return nil, err
}
return s, nil
}
One thing to note, is that you're running your function in a goroutine. Any return value inside that goroutine won't be returned to your main goroutine.
In order to get the return values for that go routine you must use channels that will wait for the values.
In your case
errChan := make(chan error)
retChan := make(chan SomeStructType)
go func() {
myVal, err := sm.MethodA()
if err != nil {
errChan <- err
return
}
if err := sm.MethodB(); err != nil {
errChan <- err
return
}
retChan <- myVal
}()
select {
case err := <-errChan:
fmt.Println(err)
case val := <-retChan:
fmt.Printf("My value: %v\n", val)
}
You can mess around with it here to make more sense out of it:
http://play.golang.org/p/TtfFIZerhk