I've created function which needs to run in goroutine, the code is working(this is just simple sample to illustrate the issue)
go rze(ftFilePath, 2)
func rze(ftDataPath,duration time.Duration) error {
}
I want to do something like this
errs := make(chan error, 1)
err := go rze(ftFilePath, 2)
if err != nil{
r <- Result{result, errs}
}
but not sure how to it, most of the examples show
how you do it when you using func
https://tour.golang.org/concurrency/5
You cannot use the return value of a function that is executed with the go keyword. Use an anonymous function instead:
errs := make(chan error, 1)
go func() {
errs <- rze(ftFilePath, 2)
}()
// later:
if err := <-errs; err != nil {
// handle error
}
You can use golang errgroup pkg.
var eg errgroup.Group
eg.Go(func() error {
return rze(ftFilePath, 2)
})
if err := g.Wait(); err != nil {
r <- Result{result, errs}
}
You can handle error from go routines using a separate channel for errors. Create a separate channel specifically for errors. Each child go routine must pass corresponding errors into this channel.
This is one of the ways how it can be done:
func main() {
type err error
items := []string{"1", "2", "3"}
ch := make(chan err, len(items))
for _, f := range items {
go func(f string) {
var e err
e = testFunc(f)
ch <- e
}(f)
}
for range items {
e := <-ch
fmt.Println("Error: ", e)
if e != nil {
// DO Something
}
}
}
func testFunc(item string) error {
if item == "1" {
return errors.New("err")
}
return nil
}
Related
I have two functions that I cannot change (see first() and second() below). They are returning some data and errors (the output data is different, but in the examples below I use (string, error) for simplicity)
I would like to run them in separate goroutines - my approach:
package main
import (
"fmt"
"os"
)
func first(name string) (string, error) {
if name == "" {
return "", fmt.Errorf("empty name is not allowed")
}
fmt.Println("processing first")
return fmt.Sprintf("First hello %s", name), nil
}
func second(name string) (string, error) {
if name == "" {
return "", fmt.Errorf("empty name is not allowed")
}
fmt.Println("processing second")
return fmt.Sprintf("Second hello %s", name), nil
}
func main() {
firstCh := make(chan string)
secondCh := make(chan string)
go func() {
defer close(firstCh)
res, err := first("one")
if err != nil {
fmt.Printf("Failed to run first: %v\n", err)
}
firstCh <- res
}()
go func() {
defer close(secondCh)
res, err := second("two")
if err != nil {
fmt.Printf("Failed to run second: %v\n", err)
}
secondCh <- res
}()
resultsOne := <-firstCh
resultsTwo := <-secondCh
// It's important for my app to do error checking and stop if errors exist.
if resultsOne == "" || resultsTwo == "" {
fmt.Println("There was an ERROR")
os.Exit(1)
}
fmt.Println("ONE:", resultsOne)
fmt.Println("TWO:", resultsTwo)
}
I believe one caveat is that resultsOne := <- firstCh blocks until first goroutine finishes, but I don't care too much about this.
Can you please confirm that my approach is good? What other approaches would be better in my situation?
The example looks mostly good. A couple improvements are:
declaring your channels as buffered
firstCh := make(chan string, 1)
secondCh := make(chan string, 1)
With unbuffered channels, send operations block (until someone receives). If your goroutine #2 is much faster than the first, it will have to wait until the first finishes as well, since you receive in sequence:
resultsOne := <-firstCh // waiting on this one first
resultsTwo := <-secondCh // sender blocked because the main thread hasn't reached this point
use "golang.org/x/sync/errgroup".Group. The program will feel "less native" but it dispenses you from managing channels by hand — which trades, in a non-contrived setting, for sync'ing writes on the results:
func main() {
var (
resultsOne string
resultsTwo string
)
g := errgroup.Group{}
g.Go(func() error {
res, err := first("one")
if err != nil {
return err
}
resultsOne = res
return nil
})
g.Go(func() error {
res, err := second("two")
if err != nil {
return err
}
resultsTwo = res
return nil
})
err := g.Wait()
// ... handle err
I have the code:
go s.addItemSync(ch, cs.ResponseQueue, user)
This calls the func:
func (s *Services) addItemSync(ch types.ChannelInsertion, statusQueueName, user string) {
//func body here
}
I would however like to do this:
if ok, err := go s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
return
}
Which would change the other func to this
func (s *Services) addItemSync(ch types.ChannelInsertion, statusQueueName, user string) (bool, error) {
}
As in, I would like to be able to declare a go func but this errors out every time. Any idea how you can declare a variable while able to call the go func ability for synchronous calls? as seen in the if ok, err := go s.addItemSync(ch, cs.ResponseQueue, user); !ok { line?
If you want to wait until a go-routine has completed, you need to return results in a channel. The basic pattern, without complicating with wait groups, etc. is:
func myFunc() {
// make a channel to receive errors
errChan := make(chan error)
// launch a go routine
go doSomething(myVar, errChan)
// block until something received on the error channel
if err := <- errChan; err != nil {
// something bad happened
}
}
// your async funciton
func doSomething(myVar interface{}, errChan chan error) {
// Do stuff
if something, err := someOtherFunc(myVar); err != nil {
errChan <- err
return
}
// all good - send nil to the error channel
errChan <- nil
}
In your case if you just want to fire off a go-routine and log if an error happens, you can use an anonymous function:
go func() {
if ok, err := s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
}
}
}()
Or if you want to wait for the result:
errChan := make(chan error)
go func() {
if ok, err := s.addItemSync(ch, cs.ResponseQueue, user); !ok {
if err != nil {
errChan <- err
return
}
}
errChan <- nil
}()
// do some other stuff while we wait...
// block until go routine returns
if err := <- errChan; err != nil {
log.Log.Error("Error adding channel", zap.Error(err))
}
Note:
Your code as written, may have unexpected results if it is possible that a response where ok == false would not return an error. If this is a concern, I would suggest creating and returning a new error for cases where !ok && err == nil
I am trying to catch errors from a group of goroutines using a channel, but the channel enters an infinite loop, starts consuming CPU.
func UnzipFile(f *bytes.Buffer, location string) error {
zipReader, err := zip.NewReader(bytes.NewReader(f.Bytes()), int64(f.Len()))
if err != nil {
return err
}
if err := os.MkdirAll(location, os.ModePerm); err != nil {
return err
}
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, errorList)
fileWaitGroup := &sync.WaitGroup{}
for _, file := range zipReader.File {
fileWaitGroup.Add(1)
go writeZipFileToLocal(file, location, errorChannel, fileWaitGroup)
}
fileWaitGroup.Wait()
close(errorChannel)
log.Println(errorList)
return nil
}
func errorChannelWatch(ch chan error, list []error) {
for {
select {
case err := <- ch:
list = append(list, err)
}
}
}
func writeZipFileToLocal(file *zip.File, location string, ch chan error, wg *sync.WaitGroup) {
defer wg.Done()
zipFilehandle, err := file.Open()
if err != nil {
ch <- err
return
}
defer zipFilehandle.Close()
if file.FileInfo().IsDir() {
if err := os.MkdirAll(filepath.Join(location, file.Name), os.ModePerm); err != nil {
ch <- err
}
return
}
localFileHandle, err := os.OpenFile(filepath.Join(location, file.Name), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, file.Mode())
if err != nil {
ch <- err
return
}
defer localFileHandle.Close()
if _, err := io.Copy(localFileHandle, zipFilehandle); err != nil {
ch <- err
return
}
ch <- fmt.Errorf("Test error")
}
So I am looping a slice of files and writing them to my disk, when there is an error I report back to the errorChannel to save that error into a slice.
I use a sync.WaitGroup to wait for all goroutines and when they are done I want to print errorList and check if there was any error during the execution.
The list is always empty, even if I add ch <- fmt.Errorf("test") at the end of writeZipFileToLocal and the channel always hangs up.
I am not sure what I am missing here.
1. For the first point, the infinite loop:
Citing from golang language spec:
A receive operation on a closed channel can always proceed
immediately, yielding the element type's zero value after any
previously sent values have been received.
So in this function
func errorChannelWatch(ch chan error, list []error) {
for {
select {
case err := <- ch:
list = append(list, err)
}
}
}
after ch gets closed this turns into an infinite loop adding nil values to list.
Try this instead:
func errorChannelWatch(ch chan error, list []error) {
for err := range ch {
list = append(list, err)
}
}
2. For the second point, why you don't see anything in your error list:
The problem is this call:
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, errorList)
Here you hand errorChannelWatch the errorList as a value. So the slice errorList will not be changed by the function. What is changed, is the underlying array, as long as the append calls don't need to allocate a new one.
To remedy the situation, either hand a slice pointer to errorChannelWatch or rewrite it as a call to a closure, capturing
errorList.
For the first proposed solution, change errorChannelWatch to
func errorChannelWatch(ch chan error, list *[]error) {
for err := range ch {
*list = append(*list, err)
}
}
and the call to
errorChannel := make(chan error)
errorList := []error{}
go errorChannelWatch(errorChannel, &errorList)
For the second proposed solution, just change the call to
errorChannel := make(chan error)
errorList := []error{}
go func() {
for err := range errorChannel {
errorList = append(errorList, err)
}
} ()
3. A minor remark:
One could think, that there is a synchronisation problem here:
fileWaitGroup.Wait()
close(errorChannel)
log.Println(errorList)
How can you be sure, that errorList isn't modified, after the call to close? One could reason, that you can't know, how many values the goroutine errorChannelWatch still has to process.
Your synchronisation seems correct to me, as you do the wg.Done()
after the send to the error channel and so all error values will
be sent, when fileWaitGroup.Wait() returns.
But that can change, if someone later adds a buffering to the error
channel or alters the code.
So I would advise to at least explain the synchronisation in a comment.
I have the following function which spins off a given amount of go routines
func (r *Runner) Execute() {
var wg sync.WaitGroup
wg.Add(len(r.pipelines))
for _, p := range r.pipelines {
go executePipeline(p, &wg)
}
wg.Wait()
errs := ....//contains list of errors reported by any/all go routines
}
I was thinking there might be some way with channels, but I can't seem to figure it out.
One way to do this is using mutexes if you can make executePipeline retuen errors:
// ...
for _, p := range r.pipelines {
go func(p pipelineType) {
if err := executePipeline(p, &wg); err != nil {
mu.Lock()
errs = append(errs, err)
mu.UnLock()
}
}(p)
}
To use a channel, you can have a separate goroutine listning for errors:
errCh := make(chan error)
go func() {
for e := range errCh {
errs = append(errs, e)
}
}
and in the Execute function, make the following changes:
// ...
wg.Add(len(r.pipelines))
for _, p := range r.pipelines {
go func(p pipelineType) {
if err := executePipeline(p, &wg); err != nil {
errCh <- err
}
}(p)
}
wg.Wait()
close(errCh)
You can always use #zerkms method listed above if the number of goroutines is not high.
instead of returning error from executePipleline and using a anonymous function wrapper, you can always make above changes within the function itself.
You can use channels as #Kaveh Shahbazian suggested:
func (r *Runner) Execute() {
pipelineChan := makePipeline(r.pipelines)
for cnt := 0; cnt < len(r.pipelines); cnt++{
//recieve from channel
p := <- pipelineChan
//do something with the result
}
}
func makePipeline(pipelines []pipelineType) <-chan pipelineType{
pipelineChan := make(chan pipelineType)
go func(){
for _, p := range pipelines {
go func(p pipelineType){
pipelineChan <- executePipeline(p)
}(p)
}
}()
return pipelineChan
}
Please see this example: https://gist.github.com/steven-ferrer/9b2eeac3eed3f7667e8976f399d0b8ad
I am trying to achieve some sort of multi-thread processing over here.
func (m *Map) Parse(mapData Node) error {
wg := &sync.WaitGroup{}
for _, node := range mapData.child {
wg.Add(1)
go parseChild(node, m, wg)
}
wg.Wait()
close(errors)
return nil
}
func parseChild(node Node, m *Map, wg *sync.WaitGroup) {
defer wg.Done()
var nodeType uint8
if err := binary.Read(node.data, binary.LittleEndian, &nodeType); err != nil {
errors <- err
}
if nodeType == OTBMNodeTowns {
for _, town := range node.child {
var nodeType uint8
if err := binary.Read(town.data, binary.LittleEndian, &nodeType); err != nil {
errors <- err
return
}
if nodeType != OTBMNodeTown {
errors <- fmt.Errorf("Parsing map towns: expected %v got %v", OTBMNodeTown, nodeType)
return
}
currentTown := Town{}
if err := binary.Read(town.data, binary.LittleEndian, ¤tTown.ID); err != nil {
errors <- err
return
} else if currentTown.Name, err = town.ReadString(); err != nil {
errors <- err
return
} else if currentTown.TemplePosition, err = town.ReadPosition(); err != nil {
errors <- err
return
}
m.Towns = append(m.Towns, currentTown)
errors <- fmt.Errorf("This should be called: %v, nodeType)
return
}
}
}
But my goroutine never sends anything to the errors channel. Seems to be that the main thread is not waiting for the goroutines to even finish
I have no idea what I am missing here. Im waiting for all routines to finish using wg.Wait but doesnt seem to be working as I think it should
And yes. the slice is populated with atleast 3 results. This is the errrors channel
var (
errors = make(chan error, 0)
)
func init() {
go errChannel()
}
func errChannel() {
for {
select {
case err := <-errors:
log.Println(err)
}
}
}