testing and mocking pubsub subscription and channel in golang - go

I am trying to learn go, I have a Java background. What I am trying to accomplish is the ability to mock pubsub and when a message is received a channel sends a message.
What I am finding hard is the fact that the method has no params. I have to Mock pubsub Receive and in that method send a mock message over a channel.
when I run
go test ./... --cover
I currently get 0%
method to test
func (s *Service) Consume() error {
subscription := os.Getenv("OUTO_PUBSUB_SUBSCRIPTION")
sub := s.Client.Subscription(subscription)
ctx := context.Background()
err := sub.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
msg.Ack()
fmt.Println(msg.Data)
s.messageChannel <- string(msg.Data)
})
if err != nil {
fmt.Println(err)
return fmt.Errorf("sub.Receive: %v", err)
}
return nil
}
pubsub test
func TestConsume(t *testing.T) {
// create channels
testChan := make(chan string, 100)
service := Service{messageChannel: testChan}
sending := service.messageChannel
sending <- "Test message"
t.Run("Test when consume runs a message is sent via channel", func(t *testing.T) {
got := <-sending
want := "Test message"
if want != got {
t.Fatalf("wanted %v, got %v", want, got)
}
})
}
I am able to get the channel test to work but I need to call the Consume method so the go framework knows I am actually testing against it.
advice?
Answer provided but run into an issue mocking Pubsub client. When implementing the mock client it just hangs and the test never finishes and when using the MockClient struct the IDE complain it is not type of pubsub.Client
type MockClient struct {
}
type MockSubscription struct {
}
func (mc MockClient) Subscription() MockSubscription {
return MockSubscription{}
}
func (ms MockSubscription) Receive(ctx context.Context, f func(context.Context, *pubsub.Message)) error {
/* create the message you're mocking you're receiving here */
/* you might have to mock the message struct and its interface if you want to validate that it has be ACKed */
msg := new(pubsub.Message)
f(ctx, msg)
// return an error or not i'll be returning nil for now
return nil
}
func TestConsume(t *testing.T) {
conn, err := grpc.Dial("myMockServerAddress", grpc.WithInsecure())
if err != nil {
// TODO: Handle the error
}
ctx := context.Background()
// Now create the pubsub client with the grpc conn
client, err := pubsub.NewClient(ctx, "mockProject", option.WithGRPCConn(conn))
// create channels
testChan := make(chan string, 100)
//when MockClient{} is here a error occurs Cannot use 'MockClient{}' (type MockClient) as the type *pubsub.Client
service := Service{messageChannel: testChan, Client: client}
sending := service.messageChannel
sending <- "Test message"
t.Run("Test when consume runs a message is sent via channel", func(t *testing.T) {
service.Consume() // run the function so you can test it
got := <-sending
want := "Test message"
if want != got {
t.Fatalf("wanted %v, got %v", want, got)
}
})
}

It's a bit tedious but you'd have to create a mock of your Service struct and it's associated interface.
Below is an example, you may have to change it to match the interface properly.
type MockClient struct {
}
type MockSubscription struct {
}
func (mc MockClient) Subscription(sub string) {
return MockSubscription{}
}
func (ms MockSubscription) Receive(ctx context.Context,f func(context.Context, *pubsub.Message)) error {
/* create the message you're mocking you're receviing here */
/* you might have to mock the message struct and its interface if you want to validate that it has be ACKed */
msg := new(pubsub.Message)
f(ctx, msg)
// return an error or not i'll be returning nil for now
return nil
}
func TestConsume(t *testing.T) {
// create channels
testChan := make(chan string, 100)
service := Service{messageChannel: testChan, Client: MockClient{}}
sending := service.messageChannel
sending <- "Test message"
t.Run("Test when consume runs a message is sent via channel", func(t *testing.T) {
service.Consume() // run the function so you can test it
got := <-sending
want := "Test message"
if want != got {
t.Fatalf("wanted %v, got %v", want, got)
}
})
}

Related

Getting values inside an unmarshalled interface

I have a websocket client that receives multiple data types. A function unmarshals json received from the server into different structs depending on the data received. The struct is then returned as an interface through a channel to my main file. Since i receive multiple data types from the server, I am not able to specify the exact return value of my parsing function.
With the data in my main file, I would like to have a way to be able to then go through the different values in the data. Since I am returning an interface, this seems impossible to do. Whenever i try to index the interface, I receive an error saying c.VALUE undefined (type interface{} has no field or method VALUE).
I feel like I'm not doing something right here. The 2 solutions I've thought about so far are:
having my channel value be a generic and my listen & JSON decoder funcs (these are all put below) all return a generic or
create an interface with methods. My channel would be of this type and again, my listen & JSON decoder funcs would return this interface.
I'm not sure if either of these ways would actually solve my issue, though. I also don't know if there is one way that would be more performant compared to other ways.
Here is my code to better understand the issue
func main() {
// check if in production or testing mode
var testing bool = true // default to testing
args := os.Args
isTesting(args, &testing, &stored_data.Base_currency)
// go routine handler
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
// set ohlc interval and pairs
OHLCinterval := 5
pairs := []string{"BTC/" + stored_data.Base_currency, "EOS/" + stored_data.Base_currency}
// create ws connections
pubSocket, err := ws_client.ConnectToServer("public", testing)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// create websocket channels
pubCh := make(chan interface{})
defer close(pubCh)
// listen to websocket connections
wg.Add(1)
go pubSocket.PubListen(ctx, &wg, pubCh, testing)
// connect to data streams
pubSocket.SubscribeToOHLC(pairs, OHLCinterval)
// listen to public socket
go func() {
for c := range pubCh {
fmt.Println(c) // This is where I would like to be able to go through my data more thoroughly
}
}()
<-comms
cancel()
wg.Wait()
}
Here is what happens in the PubListen function and my JSON decoding function
func (socket *Socket) PubListen(ctx context.Context, wg *sync.WaitGroup, ch chan interface{}, testing bool) {
defer wg.Done()
defer socket.Close()
var res interface{}
socket.OnTextMessage = func(message string, socket Socket) {
res = pubJsonDecoder(message, testing)
ch <- res
}
<-ctx.Done()
log.Println("closing public socket")
return
}
func pubJsonDecoder(response string, testing bool) interface{} {
var resp interface{}
byteResponse := []byte(response)
resp, err := ohlcResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = heartBeatResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = serverConnectionStatusResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = ohlcSubscriptionResponseDecoder(byteResponse, testing)
}
}
}
return resp
}
Thanks for any help you may have
Since you seem to control the complete list of types which can be unesrialized, you can use a type swicth :
swich v := c.(type) {
case *ohlcResponse:
// in this block, v is a *ohlcRrsponse
case *heartBeatResponse:
// in this block, v is a *heartBeatResponse
case *serverConnectionStatusResponse:
// in this block, v is a *serverConnectionStatus
case *ohlcSubscriptionResponse:
// in this block, v is a *ohlcSubscriptionResponse
default:
// choose some way to report unhandled types:
log.Fatalf("unhandled response type: %T", c)
}

golang, goroutines race condition in test

I need to subscribe to a topic(topic is a channel) before publishing to a topic, but when creating a thread I need to run go Func to keep listening to channels to process messages (for example from publish or subscribe a new subscribe )
the test works (but not every time), sometimes when I run the test it ends up posting a message on the channel (topic) before I'm listening to the topic (channel)
i have this test:
func Test_useCase_publish(t *testing.T) {
for _, tt := range tests {
tt := tt
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
tt.fields.storage = &RepositoryMock{
GetTopicFunc: func(ctx context.Context, topicName vos.TopicName) (entities.Topic, error) {
return tt.fields.topic, nil
},
}
useCase := New(tt.fields.storage)
subscribed := make(chan struct{})
go func() {
tt.fields.topic.Activate()
ch, _, err := useCase.Subscribe(tt.args.ctx, tt.args.message.TopicName)
require.NoError(t, err)
close(subscribed)
msg, ok := <-ch
if ok {
fmt.Println("msg", msg)
assert.Equal(t, tt.want, msg)
}
}()
<-subscribed
err := useCase.Publish(tt.args.ctx, tt.args.message)
assert.ErrorIs(t, err, tt.wantErr)
})
}
}
topic :
func (t Topic) Activate() {
go t.listenForSubscriptions()
go t.listenForMessages()
go t.listenForKills()
}
func (t *Topic) listenForSubscriptions() {
for newSubCh := range t.newSubCh {
t.Subscribers.Store(newSubCh.GetID(), newSubCh)
}
}
func (t *Topic) listenForKills() {
for subscriberID := range t.killSubCh {
t.Subscribers.Delete(subscriberID)
}
}
func (t *Topic) listenForMessages() {
for msg := range t.newMessageCh {
m := msg
t.Subscribers.Range(func(key, value interface{}) bool {
if key == nil || value == nil {
return false
}
if subscriber, ok := value.(Subscriber); ok {
subscriber.ReceiveMessage(m)
}
return true
})
}
func (t Topic) Dispatch(message vos.Message) {
t.newMessageCh <- message
}
func (t *Topic) listenForMessages() {
for msg := range t.newMessageCh {
m := msg
t.Subscribers.Range(func(key, value interface{}) bool {
if key == nil || value == nil {
return false
}
if subscriber, ok := value.(Subscriber); ok {
subscriber.ReceiveMessage(m)
}
return true
})
}
}
subscribe:
func (u useCase) Subscribe(ctx context.Context, topicName vos.TopicName) (chan vos.Message, vos.SubscriberID, error) {
if err := topicName.Validate(); err != nil {
return nil, "", err
}
topic, err := u.storage.GetTopic(ctx, topicName)
if err != nil {
if !errors.Is(err, entities.ErrTopicNotFound) {
return nil, "", err
}
topic, err = u.createTopic(ctx, topicName)
if err != nil {
return nil, "", err
}
subscriber := entities.NewSubscriber(topic)
subscriptionCh, id := subscriber.Subscribe()
return subscriptionCh, id, nil
}
subscriber := entities.NewSubscriber(topic)
subscriptionCh, id := subscriber.Subscribe()
return subscriptionCh, id, nil
}
func (s Subscriber) Subscribe() (chan vos.Message, vos.SubscriberID) {
s.topic.addSubscriber(s)
return s.subscriptionCh, s.GetID()
}
func (s Subscriber) ReceiveMessage(msg vos.Message) {
s.subscriptionCh <- msg
}
publisher :
func (u useCase) Publish(ctx context.Context, message vos.Message) error {
if err := message.Validate(); err != nil {
return err
}
topic, err := u.storage.GetTopic(ctx, message.TopicName)
if err != nil {
return err
}
topic.Dispatch(message)
return nil
}
when I call subscribe (I send a message to a subscribe to channel and add a subscribe to my thread) when I post a message to a topic I send a message to topic channel
Some points are missing from the code you show, such as the code for .Subscribe() and .Publish(), or how the channels are instanciated (are they buffered/unbuffered ?).
One point can be said, though :
from the looks of (t *Topic) listenForSubscriptions() : this subscribing method does not send any signal to the subscriber that it has been registered.
So my guess is : your useCase.Subscribe(...) call has the information that the created channel has been written on newSubCH, but it hasn't got the inforamtion that t.Subcribers.Store(...) has completed.
So, depending on how the goroutines are scheduled, the message sending in your test function can occur before the channel has actually been registered.
To fix this, you add something that will send a signal back to the caller. One possible way :
type subscribeReq struct{
ch chan Message
done chan struct{}
}
// turn Topic.newSubCh into a chan *subscribeReq
func (t *Topic) listenForSubscriptions() {
for req := range t.newSubCh {
t.Subscribers.Store(newSubCh.GetID(), req.ch)
close(req.done)
}
}
Another point : your test function does not check if the goroutine spun with your go func(){ ... }() call completes at all, so your unit test process may also exit before the goroutine has had the chance to execute fmt.Println(msg).
A common way to check this is to use a sync.WaitGroup :
t.Run(tt.name, func(t *testing.T) {
...
useCase := New(tt.fields.storage)
subscribed := make(chan struct{})
wg := &sync.WaitGroup{} // create a *sync.WaitGroup
wg.Add(1) // increment by 1 (you start only 1 goroutine)
go func() {
defer wg.Done() // have the goroutine call wg.Done() when returning
...
}()
// send message, check that no error occurs
wg.Wait() // block here until the goroutine has completed
})

GCP Pub/sub: using goroutines to make multiple subscriber running in one application

I have found a strange behaviour when receiving message from GCP Pub/Sub.
Following codes are how I register the subscriptions using pubsub client
gcp.go
package gcp
import (
"context"
"path"
"runtime"
"google.golang.org/api/option"
"cloud.google.com/go/pubsub"
)
// PubsubClient is the GCP pubsub service client.
var PubsubClient *pubsub.Client
// Initialize initializes GCP client service using the environment.
func Initialize(env, projectName string) error {
var err error
ctx := context.Background()
credentialOpt := option.WithCredentialsFile(getFilePathByEnv(env))
PubsubClient, err = pubsub.NewClient(ctx, projectName, credentialOpt)
return err
}
// GetTopic returns the specified topic in GCP pub/sub service and create it if it not exist.
func GetTopic(topicName string) (*pubsub.Topic, error) {
topic := PubsubClient.Topic(topicName)
ctx := context.Background()
isTopicExist, err := topic.Exists(ctx)
if err != nil {
return topic, err
}
if !isTopicExist {
ctx = context.Background()
topic, err = PubsubClient.CreateTopic(ctx, topicName)
}
return topic, err
}
// GetSubscription returns the specified subscription in GCP pub/sub service and creates it if it not exist.
func GetSubscription(subName string, topic *pubsub.Topic) (*pubsub.Subscription, error) {
sub := PubsubClient.Subscription(subName)
ctx := context.Background()
isSubExist, err := sub.Exists(ctx)
if err != nil {
return sub, err
}
if !isSubExist {
ctx = context.Background()
sub, err = PubsubClient.CreateSubscription(ctx, subName, pubsub.SubscriptionConfig{Topic: topic})
}
return sub, err
}
func getFilePathByEnv(env string) string {
_, filename, _, _ := runtime.Caller(1)
switch env {
case "local":
return path.Join(path.Dir(filename), "local.json")
case "development":
return path.Join(path.Dir(filename), "development.json")
case "staging":
return path.Join(path.Dir(filename), "staging.json")
case "production":
return path.Join(path.Dir(filename), "production.json")
default:
return path.Join(path.Dir(filename), "local.json")
}
}
main.go
package main
import (
"context"
"fmt"
"log"
"net/http"
"runtime"
"runtime/debug"
"runtime/pprof"
"time"
"rpriambudi/pubsub-receiver/gcp"
"cloud.google.com/go/pubsub"
"github.com/go-chi/chi"
)
func main() {
log.Fatal(http.ListenAndServe(":4001", Route()))
}
func Route() *chi.Mux {
InitializeSubscription()
chiRoute := chi.NewRouter()
chiRoute.Route("/api", func(r chi.Router) {
r.Get("/_count", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Number of goroutines: %v", runtime.NumGoroutine())
})
r.Get("/_stack", getStackTraceHandler)
})
return chiRoute
}
func InitializeSubscription() {
gcp.Initialize("local", "fifth-bonbon-277102")
go pubsubHandler("test-topic-1", "test-topic-1-subs")
go pubsubHandler("test-topic-2", "test-topic-2-subs")
go pubsubHandler("test-topic-3", "test-topic-3-subs")
// ....
return
}
func getStackTraceHandler(w http.ResponseWriter, r *http.Request) {
stack := debug.Stack()
w.Write(stack)
pprof.Lookup("goroutine").WriteTo(w, 2)
}
func pubsubHandler(topicID string, subscriptionID string) {
topic, err := gcp.GetTopic(topicID)
fmt.Println("topic: ", topic)
if err != nil {
fmt.Println("Failed get topic: ", err)
return
}
sub, err := gcp.GetSubscription(subscriptionID, topic)
fmt.Println("subscription: ", sub)
if err != nil {
fmt.Println("Get subscription err: ", err)
return
}
err = sub.Receive(context.Background(), func(ctx context.Context, msg *pubsub.Message) {
messageHandler(subscriptionID, ctx, msg)
})
if err != nil {
fmt.Println("receive error: ", err)
}
}
func messageHandler(subscriptionID string, ctx context.Context, msg *pubsub.Message) {
defer func() {
if r := recover(); r != nil {
fmt.Println("recovered from panic.")
msg.Ack()
}
}()
fmt.Println("message of subscription: ", subscriptionID)
fmt.Println("Message ID: ", string(msg.ID))
fmt.Println("Message received: ", string(msg.Data))
msg.Ack()
time.Sleep(10 * time.Second)
}
It works great when i just have a few of pubsubHandler inside the InitializeSubscription. But when I adding more pubsubHandler inside the initialize function (approx 10 or more handler), things starting got interesting. The ack never reach the pubsub server, making the message is simply not ack-ed (I have checked the AcknowledgeRequest in metrics explorer, and no ack request coming). Thus, the message is keep coming back to the subscriber. Also, when i restart the application, sometimes it won't receive any message, neither new or an un-acked ones.
I seems to find a workaround by set the NumGoroutines to 1 for each subscription object in the pubsubHandler function.
func pubsubHandler(topicID string, subscriptionID string) {
....
sub, err := gcp.GetSubscription(subscriptionID, topic)
....
sub.ReceiverSettings.NumGoroutines = 1
err = sub.Receive(context.Background(), func(ctx context.Context, msg *pubsub.Message) {
messageHandler(subscriptionID, ctx, msg)
})
....
}
My question is, is this an intended behaviour? What is the root cause that may lead to those unexpected behaviours? Or my implementations is simply wrong, to achieve the intended results? (multiple subscription inside one application). Or is there any best practices to follow when creating a subscription handler?
In my understanding, the Receive function from pubsub.Subscription is a blocking code natively. Hence, when I tried to run it inside a goroutines, it may lead to an unexpected side effects, especially if we're not limiting the number of goroutines that may handle the messages. Is my reasoning a valid one?
Thank you for your answers, and have a good day!
Edit 1: Updating the example to a full code, since the pubsub client is not directly imported in the main.go before.
I believe the issue might be the rate at which you are handling messages (currently 10 seconds per message). If you receive too many messages at once, your client might be overwhelmed, which will lead to a buildup of a backlog of messages.
I recommend playing around with flow control settings and increasing ReceiveSettings.NumGoroutines to something higher than the default of 10. If your publish rate is high, you could also increase MaxOutstandingMessages, or completely disable the limit by setting it to -1. This tells the client to hold onto more messages at once, a limit that is shared per Receive call.

Create wrapper for multiple instances of an Event Handler

I'm trying to deal with a problem (most probably a design one) regarding the usage of Channels and the proper handling of those.
I'm using Knative Eventing/Cloud Events to create and eventing pipeline.
I want to be able to handle different channels in order to receive events originating from different sources/methods.
In order to do so, I have the implementation that follows (code has been removed in order to be concise with detailing the issue).
I have a file1.go which defines a EventHandler struct, associated methods and a couple of exported methods (CreatePreview() and SaveAndPublish()) that are the "normal" behaviour of the app and that actually receives/deals with whatever value comes on the Channel:
type EventHandler struct {
Channel chan string
}
func (ev *EventHandler) Handle(event cloudevents.Event) {
if event.Data == nil {
(...)
}
var data string
if err := event.DataAs(&data); err != nil {
(...)
}
ev.Channel <- data
defer close(ev.Channel)
}
func (ev *EventHandler) Create(param *Element) (error) {
(...) //Unimportant code
}
func (repo *Repository) CreatePreview(param1 string, param2 string, eventHandler *EventHandler) (*pb.PreviewResponse, error) {
(...)
err := eventHandler.Create(&document)
(...)
preview := <- eventHandler.Channel
(...)
}
func (repo *Repository) SaveAndPublish(param1 string, param2 bool, eventHandler *EventHandler) (*pb.PublishResponse, error) {
(...)
err := eventHandler.Create(&documentToUpdate)
(...)
published := <- eventHandler.Channel
(...)
return repo.SomeOtherMethod(published.ID)
}
Now, on my main.go function, I have the "regular" start of a service, including a gRPC Listener, a HTTP Listener and handling of events. This is done via cmux. So here's a sample of the code (again, code simplified):
func HandlerWrapper(event cloudevents.Event) {
//TODO: HOW DO I HANDLE THIS???
}
// This approach seems to cause issues since it's always the same instance
// var (
// eventHandler = &rep.EventHandler{Channel: make(chan string)}
// )
func (service *RPCService) Start() (err error) {
port, err := strconv.Atoi(os.Getenv("LISTEN_PORT"))
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
// Create multiplexer and listener types
mux := cmux.New(lis)
grpcLis := mux.Match(cmux.HTTP2())
httpLis := mux.Match(cmux.HTTP1())
// *************
// gRPC
// *************
service.server = grpc.NewServer()
reflection.Register(service.server)
pb.RegisterStoryServiceServer(service.server, service)
// *************
// HTTP
// *************
// Declare new CloudEvents Receiver
c, err := kncloudevents.NewDefaultClient(httpLis)
if err != nil {
log.Fatal("Failed to create client, ", err)
}
// *************
// Start Listeners
// *************
// start gRPC server
go func() {
if err := service.server.Serve(grpcLis); err != nil {
log.Fatalf("failed to gRPC serve: %s", err)
}
}()
// start HTTP server
go func() {
// With this line bellow, I'd have to create a Received per eventHandler. Not cool
// log.Fatal(c.StartReceiver(context.Background(), eventHandler.Handle))
// Here we use a wrapper to deal with the event handling and have a single listener
log.Fatal(c.StartReceiver(context.Background(), HandlerWrapper))
}()
if err := mux.Serve(); err != nil {
log.Fatalf("failed to Mux serve: %s", err)
}
return
}
//CreatePreview is used to save a preview for a story
func (service *RPCService) CreatePreview(ctx context.Context, input *pb.PreviewRequest) (*pb.PreviewResponse, error){
eventHandler := &rep.EventHandler{Channel: make(chan string)}
story, err := service.repo.CreatePreview("param1", "param2", eventHandler)
if err != nil {
return nil, err
}
// Return matching the `CreatePreviewResponse` message we created in our
// protobuf definition.
return &pb.PreviewResponse{Story: story}, nil
}
// SaveAndPublish is used to save a story and publish it, returning the story saved.
func (service *RPCService) SaveAndPublish(ctx context.Context, input *pb.PublishRequest) (*pb.PublishResponse, error){
eventHandler := &rep.EventHandler{Channel: make(chan string)}
story, err := service.repo.SaveAndPublish("param1", true, eventHandler)
if err != nil {
return nil, err
}
// Return matching the `SaveAndPublishResponse` message we created in our
// protobuf definition.
return &pb.PublishResponse{Story: story}, nil
}
Now, I know that instead of having to instantiate a single, global eventHandler, in order to use the eventHandler.Handle method on c.StartReceiver() on main.go I can define a wrapper that would, maybe, contain a list of eventHandlers (the HandlerWrapper() method on main.go).
However, I do not know how I could identify which instance of an EventHandler is which and how to properly handle and route these operations, and that is my question:
How do I go about this case where I want to create a Wrapper (a single function to pass into c.StartReceive()) and then let it be handled by the correct instance of Handle()?
I hope the question is clear. I've been trying to get my head around this for a couple days already and can't figure out how to do it.
Presumably, you should be able to differentiate events by using the different sources/methods coming from that event. A quick look at the event spec shows you could split into channels based on source, for example.
The main thing I see that isn't being utilized here is the context object. It seems you could glean the source from that context. This can be seen in their hello world example (check out the receive function).
For your example:
// these are all the handlers for the different sources.
type EventHandlers map[string]CloudEventHandler
var _eventHandlerKey = "cloudEventHandlers"
func HandlerWrapper(ctx context.Context, event cloudevents.Event) {
// Get event source from event context.
src := event.Context.Source
// Then get the appropriate handler for that source (attached to context).
handler := ctx.Value(_eventHandlers).(Values)[src]
// ex: src = "/foo"
handler.SendToChannel(event)
}
func main() {
eventHandlers := make(map[string]CloudEventHandler)
// create all the channels we need, add it to the context.
for _, source := range sourceTypes { // foo bar baz
handler := NewHandler(source)
eventHandlers[source] = handler
}
// start HTTP server
go func() {
// Add the handlers to the context.
context := context.WithValue(context.Background(), _eventHandlerKey, eventHandlers)
log.Fatal(c.StartReceiver(context.Background(), HandlerWrapper))
}
}()
If there are say 3 different sources to be supported, you can use the factory pattern to instantiate those different channels and an interface that all of those implement.
// CloudEventHandler Handles sending cloud events to the proper channel for processing.
type CloudEventHandler interface {
SendToChannel(cloudEvents.Event)
}
type fooHandler struct {channel chan string}
type barHandler struct {channel chan int}
type bazHandler struct {channel chan bool}
func NewHandler(source string) CloudEventHandler {
switch source {
case "/foo":
return &fooHandler{channel: make(chan string, 2)} // buffered channel
case "/bar":
return &barHandler{channel: make(chan int, 2)}
case "/baz":
return &bazHandler{channel: make(chan bool, 2)}
}
}
func (fh *fooHandler) SendToChannel(event CloudEvents.Event) {
var data string
if err := event.DataAs(&data); err != nil {
// (...)
}
go func() {
fh.channel <- data
}()
}
func (bh *barHandler) SendToChannel(event CloudEvents.Event) {
var data int
if err := event.DataAs(&data); err != nil {
// (...)
}
go func() {
bh.channel <- data
}()
}

Does RPC have a timeout mechanism?

If RPC does not have a timeout mechanism, how do I "kill" an RPC call if it is trying to call an RPC method of a server that is closed?
You can use channels to implement a timeout pattern:
import "time"
c := make(chan error, 1)
go func() { c <- client.Call("Service", args, &result) } ()
select {
case err := <-c:
// use err and result
case <-time.After(timeoutNanoseconds):
// call timed out
}
The select will block until either client.Call returns or timeoutNanoseconds elapsed.
if you want to implement a timeout (to prevent a call from taking too long), then you'll want to change rpc.Dial for net.DialTimeout (notice they're separate packages: rpc vs net). Also be aware that the returned type isn't a client any more (as it is in the previous example); instead it is a 'connection'.
conn, err := net.DialTimeout("tcp", "localhost:8080", time.Minute)
if err != nil {
log.Fatal("dialing:", err)
}
client := rpc.NewClient(conn)
It seems the only solution for net/rpc is to close the underlying connection when you notice stuck requests. Then the client should finish pending requests with "connection broken" errors.
An alternative way is to use https://github.com/valyala/gorpc , which supports timeout RPC calls out of the box.
func (client *Client) Call(serviceMethod string, args interface{}, reply interface{}) error
Call method may block goroutine forever
Change use Go method:
func (client *Client) Go(serviceMethod string, args interface{}, reply interface{}, done chan *Call) *Call
Client example:
call := rpcClient.Go(method, args, reply, make(chan *rpc.Call, 1))
select {
case <-time.After(timeout):
log.Printf("[WARN] rpc call timeout(%v) %v => %v", timeout, rpcClient, s.RpcServer)
rpcClient.Close()
return errors.New("timeout")
case resp := <-call.Done:
if resp != nil && resp.Error != nil {
rpcClient.Close()
return resp.Error
}
Or, anno now, someone might prefer to use context instead. This also takes care of returning a proper error when timed out. (context.DeadlineExceeded)
import (
"context"
"log"
"net/rpc"
)
type Client struct {
*rpc.Client
}
// CallEx is a context aware wrapper around rpc's Client.Call()
func (c *client) CallEx(ctx context.Context, serviceMethod string, args interface{}, reply interface{}) error {
ec := make(chan error, 1)
go func() {
ec <- c.Call(serviceMethod, args, reply)
}()
select {
case err := <-ec:
return err
case <-ctx.Done():
return ctx.Err()
}
}
Invoke this with a Deadlined context:
type Args struct {
A, B int
}
func main(){
rpc, err := rpc.DialHTTP("tcp", "host")
if err != nil {
t.Fatal(err)
}
c := client{rpc}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var i int
if err := c.CallEx(ctx, "Calc.Multiply", Args{2, 2}, &i); err != nil {
log.Fatal(err)
}
}

Resources