Daisy chain input,output channels together in golang - go

I have the following interface and struct
type PiplineStep interface {
Do(ctx context.Context, in <-chan Message) (<-chan Message, <-chan error, error)
}
type Pipline struct {
Steps []core.PiplineStep
}
Now I am trying to daisy the interfaces to create a pipeline like the following
for _, step := range p.Steps {
out, errc, err := step.Do(ctx, out)
errcList = append(errcList, errc)
if err != nil {
errc <- err
return
}
select {
case outer <- msg:
case <-ctx.Done():
return
}
}
But the compiler says no is this possible?
I get the following Error 'out declared and not used' i have attempted following but it appears that all steps are receiving the same chan
for _, step := range p.Steps {
var tmpOut <-chan core.Message
tmpOut = out
tmpOut, errcTmp, err := step.Do(ctx, tmpOut)
errcList = append(errcList, errcTmp)
if err != nil {
errc <- err
return
}
select {
case out <- msg:
case <-ctx.Done():
return
}
}

You have to declare your channel variable outside the loop if you want to re-use it in each iteration (errors and context omitted for brevity):
package main
import "fmt"
func main() {
var pipeline Pipeline
pipeline.Steps = append(pipeline.Steps,
AddBang{},
AddBang{},
AddBang{},
)
src := make(chan Message)
pipe := src
for _, s := range pipeline.Steps {
pipe = s.Do(pipe)
}
go func() {
src <- "msg 1"
src <- "msg 2"
src <- "msg 3"
}()
fmt.Println(<-pipe)
fmt.Println(<-pipe)
fmt.Println(<-pipe)
}
type Message string
type Pipeline struct {
Steps []PipelineStep
}
type PipelineStep interface {
Do(in chan Message) chan Message
}
type AddBang struct{}
func (AddBang) Do(in chan Message) chan Message {
out := make(chan Message)
go func() {
defer close(out)
for m := range in {
out <- m + "!"
}
}()
return out
}
Try it on the playground: https://play.golang.org/p/ItVLUBRpNA1

Related

How can I test method in proper way in my case?

I've written this code for education purposes, now I need to write some test for it.
First, I need to test method Worker, but I don't understand how can I do it properly?
I'm totally new to test and to Go generally.
package worker
import (
"context"
"fmt"
)
type Queue interface {
TakeMessage() (<-chan string, error)
}
type Download interface {
Download(url string) error
}
type Worker struct {
queue Queue
download Download
}
func NewWorker(queue Queue, download Download) *Worker {
newWorker := Worker{}
newWorker.queue = queue
newWorker.download = download
return &newWorker
}
func (w *Worker) Worker(ctx context.Context) error {
msgs, err := w.queue.TakeMessage()
if err != nil {
return fmt.Errorf("error while consume queue: %w", err)
}
for {
select {
case <-ctx.Done():
return nil
case msg := <-msgs:
fmt.Println(msg)
w.download.Download(msg)
}
}
}
I write some lines of testing code and I suppossed to check if Worker return nil when ctx.Done() is passed to channel Now this test doesn't work, it's stuck maybe because of infinite cycle in worker method. Then I need to check if method takes messages from queue and pass it to Download method.
package worker
import (
"context"
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
)
type MockQueue struct {
mock.Mock
}
type MockDownloader struct {
mock.Mock
}
func (m *MockQueue) TakeMessage() (<-chan string, error) {
strCh := make(chan string)
strCh <- "some_url/some.txt"
return strCh, nil
}
func (d *MockDownloader) Download(url string) error {
if url == "some_url/some.txt" {
return nil
} else {
return errors.New(url)
}
}
func TestWorkerCloseContext(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
resultCh := make(chan error)
newQueue := &MockQueue{}
newDownload := &MockDownloader{}
newWorker := Worker{newQueue, newDownload}
go func() {
resultCh <- newWorker.Worker(ctx)
}()
cancel()
assert.Nil(t, <-resultCh)
}
func TestWorkerMessageReceive(t \*testing.T) {
ctx, cancel := context.WithCancel(context.Background())
resultCh := make(chan error)
newQueue := &MockQueue{}
newDownload := &MockDownloader{}
newWorker := Worker{newQueue, newDownload}
go func() {
resultCh \<- newWorker.Worker(ctx)
}()
//some code here
}
Go comes with a pretty extensive built-in testing library. See https://pkg.go.dev/testing
I would recommend looking at the below tutorials:
https://go.dev/blog/cover
https://blog.alexellis.io/golang-writing-unit-tests/
https://dev.to/quii/learn-go-by-writing-tests-structs-methods-interfaces--table-driven-tests-1p01

golang, goroutines race condition in test

I need to subscribe to a topic(topic is a channel) before publishing to a topic, but when creating a thread I need to run go Func to keep listening to channels to process messages (for example from publish or subscribe a new subscribe )
the test works (but not every time), sometimes when I run the test it ends up posting a message on the channel (topic) before I'm listening to the topic (channel)
i have this test:
func Test_useCase_publish(t *testing.T) {
for _, tt := range tests {
tt := tt
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
tt.fields.storage = &RepositoryMock{
GetTopicFunc: func(ctx context.Context, topicName vos.TopicName) (entities.Topic, error) {
return tt.fields.topic, nil
},
}
useCase := New(tt.fields.storage)
subscribed := make(chan struct{})
go func() {
tt.fields.topic.Activate()
ch, _, err := useCase.Subscribe(tt.args.ctx, tt.args.message.TopicName)
require.NoError(t, err)
close(subscribed)
msg, ok := <-ch
if ok {
fmt.Println("msg", msg)
assert.Equal(t, tt.want, msg)
}
}()
<-subscribed
err := useCase.Publish(tt.args.ctx, tt.args.message)
assert.ErrorIs(t, err, tt.wantErr)
})
}
}
topic :
func (t Topic) Activate() {
go t.listenForSubscriptions()
go t.listenForMessages()
go t.listenForKills()
}
func (t *Topic) listenForSubscriptions() {
for newSubCh := range t.newSubCh {
t.Subscribers.Store(newSubCh.GetID(), newSubCh)
}
}
func (t *Topic) listenForKills() {
for subscriberID := range t.killSubCh {
t.Subscribers.Delete(subscriberID)
}
}
func (t *Topic) listenForMessages() {
for msg := range t.newMessageCh {
m := msg
t.Subscribers.Range(func(key, value interface{}) bool {
if key == nil || value == nil {
return false
}
if subscriber, ok := value.(Subscriber); ok {
subscriber.ReceiveMessage(m)
}
return true
})
}
func (t Topic) Dispatch(message vos.Message) {
t.newMessageCh <- message
}
func (t *Topic) listenForMessages() {
for msg := range t.newMessageCh {
m := msg
t.Subscribers.Range(func(key, value interface{}) bool {
if key == nil || value == nil {
return false
}
if subscriber, ok := value.(Subscriber); ok {
subscriber.ReceiveMessage(m)
}
return true
})
}
}
subscribe:
func (u useCase) Subscribe(ctx context.Context, topicName vos.TopicName) (chan vos.Message, vos.SubscriberID, error) {
if err := topicName.Validate(); err != nil {
return nil, "", err
}
topic, err := u.storage.GetTopic(ctx, topicName)
if err != nil {
if !errors.Is(err, entities.ErrTopicNotFound) {
return nil, "", err
}
topic, err = u.createTopic(ctx, topicName)
if err != nil {
return nil, "", err
}
subscriber := entities.NewSubscriber(topic)
subscriptionCh, id := subscriber.Subscribe()
return subscriptionCh, id, nil
}
subscriber := entities.NewSubscriber(topic)
subscriptionCh, id := subscriber.Subscribe()
return subscriptionCh, id, nil
}
func (s Subscriber) Subscribe() (chan vos.Message, vos.SubscriberID) {
s.topic.addSubscriber(s)
return s.subscriptionCh, s.GetID()
}
func (s Subscriber) ReceiveMessage(msg vos.Message) {
s.subscriptionCh <- msg
}
publisher :
func (u useCase) Publish(ctx context.Context, message vos.Message) error {
if err := message.Validate(); err != nil {
return err
}
topic, err := u.storage.GetTopic(ctx, message.TopicName)
if err != nil {
return err
}
topic.Dispatch(message)
return nil
}
when I call subscribe (I send a message to a subscribe to channel and add a subscribe to my thread) when I post a message to a topic I send a message to topic channel
Some points are missing from the code you show, such as the code for .Subscribe() and .Publish(), or how the channels are instanciated (are they buffered/unbuffered ?).
One point can be said, though :
from the looks of (t *Topic) listenForSubscriptions() : this subscribing method does not send any signal to the subscriber that it has been registered.
So my guess is : your useCase.Subscribe(...) call has the information that the created channel has been written on newSubCH, but it hasn't got the inforamtion that t.Subcribers.Store(...) has completed.
So, depending on how the goroutines are scheduled, the message sending in your test function can occur before the channel has actually been registered.
To fix this, you add something that will send a signal back to the caller. One possible way :
type subscribeReq struct{
ch chan Message
done chan struct{}
}
// turn Topic.newSubCh into a chan *subscribeReq
func (t *Topic) listenForSubscriptions() {
for req := range t.newSubCh {
t.Subscribers.Store(newSubCh.GetID(), req.ch)
close(req.done)
}
}
Another point : your test function does not check if the goroutine spun with your go func(){ ... }() call completes at all, so your unit test process may also exit before the goroutine has had the chance to execute fmt.Println(msg).
A common way to check this is to use a sync.WaitGroup :
t.Run(tt.name, func(t *testing.T) {
...
useCase := New(tt.fields.storage)
subscribed := make(chan struct{})
wg := &sync.WaitGroup{} // create a *sync.WaitGroup
wg.Add(1) // increment by 1 (you start only 1 goroutine)
go func() {
defer wg.Done() // have the goroutine call wg.Done() when returning
...
}()
// send message, check that no error occurs
wg.Wait() // block here until the goroutine has completed
})

Best time to close channel, when iterating over channel

I am playing around with Golang and I created this little app to make several concurrent api calls using goroutines.
While the app works, after the calls complete, the app gets stuck, which makes sense because it cannot exit the range c loop because the channel is not closed.
I am not sure where to better close the channel in this pattern.
package main
import "fmt"
import "net/http"
func main() {
links := []string{
"https://github.com/fabpot",
"https://github.com/andrew",
"https://github.com/taylorotwell",
"https://github.com/egoist",
"https://github.com/HugoGiraudel",
}
checkUrls(links)
}
func checkUrls(urls []string) {
c := make(chan string)
for _, link := range urls {
go checkUrl(link, c)
}
for msg := range c {
fmt.Println(msg)
}
close(c) //this won't get hit
}
func checkUrl(url string, c chan string) {
_, err := http.Get(url)
if err != nil {
c <- "We could not reach:" + url
} else {
c <- "Success reaching the website:" + url
}
}
You close a channel when there are no more values to send, so in this case it's when all checkUrl goroutines have completed.
var wg sync.WaitGroup
func checkUrls(urls []string) {
c := make(chan string)
for _, link := range urls {
wg.Add(1)
go checkUrl(link, c)
}
go func() {
wg.Wait()
close(c)
}()
for msg := range c {
fmt.Println(msg)
}
}
func checkUrl(url string, c chan string) {
defer wg.Done()
_, err := http.Get(url)
if err != nil {
c <- "We could not reach:" + url
} else {
c <- "Success reaching the website:" + url
}
}
(Note that the error from http.Get is only going to reflect connection and protocol errors. It is not going to contain http server errors if you're expecting those too, which you must be seeing how you're checking for paths and not just hosts.)
When writing programs in Go using channels and goroutines always think about who (which function) owns a channel. I prefer the practice of letting the function who owns a channel close it. If i were to write this i would do as shown below.
Note: A better way to handle situations like this is the Fan-out, fan-in concurrency pattern. refer(https://blog.golang.org/pipelines)Go Concurrency Patterns
package main
import "fmt"
import "net/http"
import "sync"
func main() {
links := []string{
"https://github.com/fabpot",
"https://github.com/andrew",
"https://github.com/taylorotwell",
"https://github.com/egoist",
"https://github.com/HugoGiraudel",
}
processURLS(links)
fmt.Println("End of Main")
}
func processURLS(links []string) {
resultsChan := checkUrls(links)
for msg := range resultsChan {
fmt.Println(msg)
}
}
func checkUrls(urls []string) chan string {
outChan := make(chan string)
go func(urls []string) {
defer close(outChan)
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go checkUrl(&wg, url, outChan)
}
wg.Wait()
}(urls)
return outChan
}
func checkUrl(wg *sync.WaitGroup, url string, c chan string) {
defer wg.Done()
_, err := http.Get(url)
if err != nil {
c <- "We could not reach:" + url
} else {
c <- "Success reaching the website:" + url
}
}

Get responses from multiple go routines into an array

I need to fetch responses from multiple go routines and put them into an array. I know that channels could be used for this, however I am not sure how I can make sure that all go routines have finished processing the results. Thus I am using a waitgroup.
Code
func main() {
log.Info("Collecting ints")
var results []int32
for _, broker := range e.BrokersByBrokerID {
wg.Add(1)
go getInt32(&wg)
}
wg.Wait()
log.info("Collected")
}
func getInt32(wg *sync.WaitGroup) (int32, error) {
defer wg.Done()
// Just to show that this method may just return an error and no int32
err := broker.Open(config)
if err != nil && err != sarama.ErrAlreadyConnected {
return 0, fmt.Errorf("Cannot connect to broker '%v': %s", broker.ID(), err)
}
defer broker.Close()
return 1003, nil
}
My question
How can I put all the response int32 (which may return an error) into my int32 array, making sure that all go routines have finished their processing work and returned either the error or the int?
If you don't process the return values of the function launched as a goroutine, they are discarded. See What happens to return value from goroutine.
You may use a slice to collect the results, where each goroutine could receive the index to put the results to, or alternatively the address of the element. See Can I concurrently write different slice elements. Note that if you use this, the slice must be pre-allocated and only the element belonging to the goroutine may be written, you can't "touch" other elements and you can't append to the slice.
Or you may use a channel, on which the goroutines send values that include the index or ID of the item they processed, so the collecting goroutine can identify or order them. See How to collect values from N goroutines executed in a specific order?
If processing should stop on the first error encountered, see Close multiple goroutine if an error occurs in one in go
Here's an example how it could look like when using a channel. Note that no waitgroup is needed here, because we know that we expect as many values on the channel as many goroutines we launch.
type result struct {
task int32
data int32
err error
}
func main() {
tasks := []int32{1, 2, 3, 4}
ch := make(chan result)
for _, task := range tasks {
go calcTask(task, ch)
}
// Collect results:
results := make([]result, len(tasks))
for i := range results {
results[i] = <-ch
}
fmt.Printf("Results: %+v\n", results)
}
func calcTask(task int32, ch chan<- result) {
if task > 2 {
// Simulate failure
ch <- result{task: task, err: fmt.Errorf("task %v failed", task)}
return
}
// Simulate success
ch <- result{task: task, data: task * 2, err: nil}
}
Output (try ot on the Go Playground):
Results: [{task:4 data:0 err:0x40e130} {task:1 data:2 err:<nil>} {task:2 data:4 err:<nil>} {task:3 data:0 err:0x40e138}]
I also believe you have to use channel, it must be something like this:
package main
import (
"fmt"
"log"
"sync"
)
var (
BrokersByBrokerID = []int32{1, 2, 3}
)
type result struct {
data string
err string // you must use error type here
}
func main() {
var wg sync.WaitGroup
var results []result
ch := make(chan result)
for _, broker := range BrokersByBrokerID {
wg.Add(1)
go getInt32(ch, &wg, broker)
}
go func() {
for v := range ch {
results = append(results, v)
}
}()
wg.Wait()
close(ch)
log.Printf("collected %v", results)
}
func getInt32(ch chan result, wg *sync.WaitGroup, broker int32) {
defer wg.Done()
if broker == 1 {
ch <- result{err: fmt.Sprintf("error: gor broker 1")}
return
}
ch <- result{data: fmt.Sprintf("broker %d - ok", broker)}
}
Result will look like this:
2019/02/05 15:26:28 collected [{broker 3 - ok } {broker 2 - ok } { error: gor broker 1}]
package main
import (
"fmt"
"log"
"sync"
)
var (
BrokersByBrokerID = []int{1, 2, 3, 4}
)
type result struct {
data string
err string // you must use error type here
}
func main() {
var wg sync.WaitGroup
var results []int
ch := make(chan int)
done := make(chan bool)
for _, broker := range BrokersByBrokerID {
wg.Add(1)
go func(i int) {
defer wg.Done()
ch <- i
if i == 4 {
done <- true
}
}(broker)
}
L:
for {
select {
case v := <-ch:
results = append(results, v)
if len(results) == 4 {
//<-done
close(ch)
break L
}
case _ = <-done:
break
}
}
fmt.Println("STOPPED")
//<-done
wg.Wait()
log.Printf("collected %v", results)
}
Thank cn007b and Edenshaw. My answer is based on their answers.
As Edenshaw commented, need another sync.Waitgroup for goroutine which getting results from channel, or you may get an incomplete array.
package main
import (
"fmt"
"sync"
"encoding/json"
)
type Resp struct {
id int
}
func main() {
var wg sync.WaitGroup
chanRes := make(chan interface{}, 3)
for i := 0; i < 3; i++ {
wg.Add(1)
resp := &Resp{}
go func(i int, resp *Resp) {
defer wg.Done()
resp.id = i
chanRes <- resp
}(i, resp)
}
res := make([]interface{}, 0)
var wg2 sync.WaitGroup
wg2.Add(1)
go func() {
defer wg2.Done()
for v := range chanRes {
res = append(res, v.(*Resp).id)
}
}()
wg.Wait()
close(chanRes)
wg2.Wait()
resStr, _ := json.Marshal(res)
fmt.Println(string(resStr))
}
package main
import (
"fmt"
"log"
"sync"
"time"
)
var (
BrokersByBrokerID = []int{1, 2, 3, 4}
)
type result struct {
data string
err string // you must use error type here
}
func main() {
var wg sync.WaitGroup.
var results []int
ch := make(chan int)
done := make(chan bool)
for _, broker := range BrokersByBrokerID {
wg.Add(1)
go func(i int) {
defer wg.Done()
ch <- i
if i == 4 {
done <- true
}
}(broker)
}
for v := range ch {
results = append(results, v)
if len(results) == 4 {
close(ch)
}
}
fmt.Println("STOPPED")
<-done
wg.Wait()
log.Printf("collected %v", results)
}
</pre>

How to return a error from a goroutine (through channels)

When i write a function in Go, it should return a value and an error like
func createHashedPassword(password string) (string, error) {
//code
}
I want to execute this createHashedPassword in a goroutine and I think to pass data via channel.
But my question is, how can I handle error here or in the goroutine?
It's common to bundle multiple outputs into a struct, and return them together over a single channel.
type Result struct {
Message string
Error error
}
ch := make(chan Result)
You can pass in an error channel as well as a result channel.
errors := make(chan error, 0)
results := make(chan string, 0)
password := "test"
go func() {
result, err := createHashedPassword(string password)
if err != nil {
errors <- err
return
}
results <- result
}()
// do something else
// When you are ready to read from goroutine do this:
select {
case err := <- errors:
println(err)
case res := <- results:
println(res)
}
Here are my two preferred ways:
Two channels, wrapped
This is the "two channels" way, but wrapped into a function to make it look similar to the common pattern:
func createHashedPasswordAsynchronously(password string) (chan string, chan error) {
resultCh := make(chan string)
errorCh := make(chan error)
go func(password string) {
//code
if err != nil {
errorCh <- errors.New("Does not compute")
} else {
resultCh <- "8badf00d"
}
}(password)
return resultCh, errorCh
}
And called like this:
resultCh, errorCh := createHashedPasswordAsynchronously("mysecret")
select {
case result := <-resultCh:
storeHashedPassword(result)
case err := <-errorCh:
log.Println(err.Error())
}
Anonymous struct
This is the "anonymous struct" way, similar to #saward's answer, but without naming the struct members explicitly:
go func(password string, ch chan struct {
string
error
}) {
//code
if err != nil {
ch <- struct {
string
error
}{"", errors.New("Does not compute")}
} else {
ch <- struct {
string
error
}{"8badf00d", nil}
}
}("mysecret", ch)
r := <-ch
if r.error != nil {
log.Println(r.error.Error())
} else {
storeHashedPassword(r.string)
}
(since I cannot comment yet...)
I echo what JimB said with:
type Result struct {
Message string
Error error
}
ch := make(chan Result)
The trouble with two separate channels, one for the result, and another for the error, is that (as I understand) it won't support concurrent threads out of the box.
You could, for example, have two threads sending data at the same time, where the replies get out of order. That is, you receive the result from thread 1 first, but the error from thread 2 first.
It's easy to create new types like JimB suggested, and should work well with goroutines.
One of the common patterns is to pass a error channel and a result channel to the function:
package main
import (
"fmt"
)
func main() {
password := "blah blah"
resultChan := make(chan string)
errChan := make(chan error)
go createHashedPassword(password, resultChan, errChan)
// Do some other stuff
select {
case err := <-errChan:
fmt.Println(err)
case res := <-resultChan:
fmt.Println(res)
}
}
func createHashedPassword(password string, resultChan chan string, errChan chan error) {
// your code for hashing and stuff
if err != nil {
errChan <- err
return
}
resultChan <- result
}
You can also make channels inside the createHashedPassword as AndreKR said,
or call the createHashedPassword inside an anonymous goroutine as Ian Davis said.
There is also some other possible patterns, and you can combine any of them with combined struct:
type Result struct {
err error
msg string
}
or
type Result struct {
err error
msg *string // in cases that empty string can be a true result, you can evaluate msg with nil
}
I should also mention that in some special cases, we don't return the error and just log it there.
package main
import (
"fmt"
"log"
)
func main() {
password := "blah blah"
resultChan := make(chan string)
go createHashedPassword(password, resultChan)
// Do some other stuff
fmt.Println(<-resultChan) // Be careful that it's a blocking code.
}
func createHashedPassword(password string, resultChan chan string) {
// your code for hashing and stuff
if err != nil {
log.Println(err) // *** Here we just log the error, or handle it.
}
resultChan <- result
}

Resources