Getting values inside an unmarshalled interface - go

I have a websocket client that receives multiple data types. A function unmarshals json received from the server into different structs depending on the data received. The struct is then returned as an interface through a channel to my main file. Since i receive multiple data types from the server, I am not able to specify the exact return value of my parsing function.
With the data in my main file, I would like to have a way to be able to then go through the different values in the data. Since I am returning an interface, this seems impossible to do. Whenever i try to index the interface, I receive an error saying c.VALUE undefined (type interface{} has no field or method VALUE).
I feel like I'm not doing something right here. The 2 solutions I've thought about so far are:
having my channel value be a generic and my listen & JSON decoder funcs (these are all put below) all return a generic or
create an interface with methods. My channel would be of this type and again, my listen & JSON decoder funcs would return this interface.
I'm not sure if either of these ways would actually solve my issue, though. I also don't know if there is one way that would be more performant compared to other ways.
Here is my code to better understand the issue
func main() {
// check if in production or testing mode
var testing bool = true // default to testing
args := os.Args
isTesting(args, &testing, &stored_data.Base_currency)
// go routine handler
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
// set ohlc interval and pairs
OHLCinterval := 5
pairs := []string{"BTC/" + stored_data.Base_currency, "EOS/" + stored_data.Base_currency}
// create ws connections
pubSocket, err := ws_client.ConnectToServer("public", testing)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// create websocket channels
pubCh := make(chan interface{})
defer close(pubCh)
// listen to websocket connections
wg.Add(1)
go pubSocket.PubListen(ctx, &wg, pubCh, testing)
// connect to data streams
pubSocket.SubscribeToOHLC(pairs, OHLCinterval)
// listen to public socket
go func() {
for c := range pubCh {
fmt.Println(c) // This is where I would like to be able to go through my data more thoroughly
}
}()
<-comms
cancel()
wg.Wait()
}
Here is what happens in the PubListen function and my JSON decoding function
func (socket *Socket) PubListen(ctx context.Context, wg *sync.WaitGroup, ch chan interface{}, testing bool) {
defer wg.Done()
defer socket.Close()
var res interface{}
socket.OnTextMessage = func(message string, socket Socket) {
res = pubJsonDecoder(message, testing)
ch <- res
}
<-ctx.Done()
log.Println("closing public socket")
return
}
func pubJsonDecoder(response string, testing bool) interface{} {
var resp interface{}
byteResponse := []byte(response)
resp, err := ohlcResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = heartBeatResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = serverConnectionStatusResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = ohlcSubscriptionResponseDecoder(byteResponse, testing)
}
}
}
return resp
}
Thanks for any help you may have

Since you seem to control the complete list of types which can be unesrialized, you can use a type swicth :
swich v := c.(type) {
case *ohlcResponse:
// in this block, v is a *ohlcRrsponse
case *heartBeatResponse:
// in this block, v is a *heartBeatResponse
case *serverConnectionStatusResponse:
// in this block, v is a *serverConnectionStatus
case *ohlcSubscriptionResponse:
// in this block, v is a *ohlcSubscriptionResponse
default:
// choose some way to report unhandled types:
log.Fatalf("unhandled response type: %T", c)
}

Related

Closing Channel Inside of a Struct

I've been trying to look through existing posts about closing a channel, but I can't seem to find exactly what I'm looking for. I have two channels inside of a struct. After making the channels, I tried to run defer close(channelName) immediately after, but the channels immediately closed and I got an error. If I don't deal with closing the channels, the code runs fine, but doesn't shut down gracefully and I have to use ctrl z to suspend the program, since it's still running after using ctrl c. Here are the important parts of the code:
type WebsocketClient struct {
pubSocket ws_client.Socket
privSocket ws_client.Socket
pubChan chan interface{}
privChan chan interface{}
}
type KrakenClient struct {
WebSocket WebsocketClient
Testing bool
}
func (client *KrakenClient) initChannels() {
client.WebSocket.pubChan = make(chan interface{})
client.WebSocket.privChan = make(chan interface{})
//defer close(client.WebSocket.pubChan)
//defer close(client.WebSocket.privChan)
}
func (client *KrakenClient) InitWebSocketClient(wg *sync.WaitGroup, testing bool) {
client.initTesting(testing)
client.initChannels()
client.startWebSocketConnection(wg)
}
func (client *KrakenClient) PubDecoder(wg *sync.WaitGroup, ctx context.Context) {
wg.Add(1)
defer wg.Done()
defer client.WebSocket.pubSocket.Close()
if err := PubSocketGuard(client.WebSocket); err != nil { // guard clause checker. makes sure i'm actually using a public WebSocket
panic(err)
}
var res interface{}
ws_client.ReceiveLocker(&client.WebSocket.pubSocket)
client.WebSocket.pubSocket.OnTextMessage = func(message string, socket ws_client.Socket) {
res = ws_client.PubJsonDecoder(message, client.Testing)
client.WebSocket.pubChan <- res
}
ws_client.ReceiveUnlocker(&client.WebSocket.pubSocket)
<-ctx.Done()
log.Println("closing public socket")
return
}
func (client *KrakenClient) PubListen(wg *sync.WaitGroup, ctx context.Context, ohlcMap *types.OHLCVals) {
wg.Add(1)
defer wg.Done()
for c := range client.WebSocket.pubChan {
switch v := c.(type) {
// More code here. Not important to the channel problem
}
<-ctx.Done()
}
func main() {
var testing bool = true
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
kraken := &kraken_client.KrakenClient{}
kraken.InitWebSocketClient(&wg, testing)
go kraken.PubDecoder(&wg, ctx)
kraken.SubscribeToOHLC(&wg, []string{"BTC/USD"}, 5)
ohlcMap := types.OHLCVals{}
go kraken.PubListen(&wg, ctx, &ohlcMap)
<-comms
cancel()
wg.Wait()
}
In summary, struct KrakenClient has a type WebsocketClient, which holds 2 WebSockets and 2 channels. Calling the InitWebSocketClient func on the KrakenClient struct creates the two channels and connects to the WS server. Once connected to the server, I start to unmarshal all responses and send those responses to a channel. I then subscribe to a specific endpoint, and start to "listen" to the responses (actually go through the data and add it to the ohlcMap variable that's passed to the listen func depending on certain circumstances). I just don't understand where I should be closing the channels. Do I need to create a Close function on my KrakenClient struct that defers when the channels are closed? If so, where would it even go? Appreciate any help!
You must call defer in main func because when you call defer in initChannels func it's immediately close your channels after making.
The defer always execute when the function (initChannels in your code) has been return.
you can write a closing function for close the channels and call it on main func like as:
func (client *KrakenClient) closeChannels() {
close(client.WebSocket.pubChan)
close(client.WebSocket.privChan)
}
func (client *KrakenClient) initChannels() {
client.WebSocket.pubChan = make(chan interface{})
client.WebSocket.privChan = make(chan interface{})
}
func main() {
...
kraken.InitWebSocketClient(&wg, testing)
defer kraken.closeChannels()
...
}

Golang Concurrency Issue to introduce timeout

I wish to implement parallel api calling in golang using go routines. Once the requests are fired,
I need to wait for all responses (which take different time).
If any of the request fails and returns an error, I wish to end (or pretend) the routines.
I also want to have a timeout value associated with each go routine (or api call).
I have implemented the below for 1 and 2, but need help as to how can I implement 3. Also, feedback on 1 and 2 will also help.
package main
import (
"errors"
"fmt"
"sync"
"time"
)
func main() {
var wg sync.WaitGroup
c := make(chan interface{}, 1)
c2 := make(chan interface{}, 1)
err := make(chan interface{})
wg.Add(1)
go func() {
defer wg.Done()
result, e := doSomeWork()
if e != nil {
err <- e
return
}
c <- result
}()
wg.Add(1)
go func() {
defer wg.Done()
result2, e := doSomeWork2()
if e != nil {
err <- e
return
}
c2 <- result2
}()
go func() {
wg.Wait()
close(c)
close(c2)
close(err)
}()
for e := range err {
// here error happend u could exit your caller function
fmt.Println("Error==>", e)
return
}
fmt.Println(<-c, <-c2)
}
// mimic api call 1
func doSomeWork() (function1, error) {
time.Sleep(10 * time.Second)
obj := function1{"ABC", "29"}
return obj, nil
}
type function1 struct {
Name string
Age string
}
// mimic api call 2
func doSomeWork2() (function2, error) {
time.Sleep(4 * time.Second)
r := errors.New("Error Occured")
if 1 == 2 {
fmt.Println(r)
}
obj := function2{"Delhi", "Delhi"}
// return error as nil for now
return obj, nil
}
type function2 struct {
City string
State string
}
Thanks in advance.
This kind of fork-and-join pattern is exactly what golang.org/x/sync/errgroup was designed for. (Identifying the appropriate “first error” from a group of goroutines can be surprisingly subtle.)
You can use errgroup.WithContext to obtain a context.Context that is cancelled if any of the goroutines in the group returns. The (*Group).Wait method waits for the goroutines to complete and returns the first error.
For your example, that might look something like: https://play.golang.org/p/jqYeb4chHCZ.
You can then inject a timeout within any given call by wrapping the Context using context.WithTimeout.
(However, in my experience if you've plumbed in cancellation correctly, explicit timeouts are almost never helpful — the end user can cancel explicitly if they get tired of waiting, and you probably don't want to promote degraded service to a complete outage if something starts to take just a bit longer than you expected.)
To support timeouts and cancelation of goroutine work, the standard mechanism is to use context.Context.
ctx := context.Background() // root context
// wrap the context with a timeout and/or cancelation mechanism
ctx, cancel := context.WithTimeout(ctx, 5*time.Second) // with timeout or cancel
//ctx, cancel := context.WithCancel(ctx) // no timeout just cancel
defer cancel() // avoid memory leak if we never cancel/timeout
Next your worker goroutines need to support taking and monitoring the state of the ctx. To do this in parallel with the time.Sleep (to mimic a long computation), convert the sleep to a channel based solution:
// mimic api call 1
func doSomeWork(ctx context.Context) (function1, error) {
//time.Sleep(10 * time.Second)
select {
case <-time.After(10 * time.Second):
// wait completed
case <-ctx.Done():
return function1{}, ctx.Err()
}
// ...
}
And if one worker goroutine fails, to signal to the other worker that the request should be aborted, simply call the cancel() function.
result, e := doSomeWork(ctx)
if e != nil {
cancel() // <- add this
err <- e
return
}
Pulling this all together:
https://play.golang.org/p/1Kpe_tre7XI
EDIT: the sleep example above is obviously a contrived example of how to abort a "fake" task. In the real world, http or SQL DB calls would be involve - and since go 1.7 & 1.8 - the standard library added context support to any of these potentially blocking calls:
func doSomeWork(ctx context.Context) (error)
// DB
db, err := sql.Open("mysql", "...") // check err
//rows, err := db.Query("SELECT age from users", age)
rows, err := db.QueryContext(ctx, "SELECT age from users", age)
if err != nil {
return err // will return with error if context is canceled
}
// http
// req, err := http.NewRequest("GET", "http://example.com", nil)
req, err := http.NewRequestWithContext(ctx, "GET", "http://example.com", nil) // check err
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err // will return with error if context is canceled
}
}
EDIT (2): to poll a context's state without blocking, leverage select's default branch:
select {
case <-ctx.Done():
return ctx.Err()
default:
// if ctx is not done - this branch is used
}
the default branch can optional have code in it, but even if it is empty of code it's presence will prevent blocking - and thus just poll the status of the context in that instant of time.

Create wrapper for multiple instances of an Event Handler

I'm trying to deal with a problem (most probably a design one) regarding the usage of Channels and the proper handling of those.
I'm using Knative Eventing/Cloud Events to create and eventing pipeline.
I want to be able to handle different channels in order to receive events originating from different sources/methods.
In order to do so, I have the implementation that follows (code has been removed in order to be concise with detailing the issue).
I have a file1.go which defines a EventHandler struct, associated methods and a couple of exported methods (CreatePreview() and SaveAndPublish()) that are the "normal" behaviour of the app and that actually receives/deals with whatever value comes on the Channel:
type EventHandler struct {
Channel chan string
}
func (ev *EventHandler) Handle(event cloudevents.Event) {
if event.Data == nil {
(...)
}
var data string
if err := event.DataAs(&data); err != nil {
(...)
}
ev.Channel <- data
defer close(ev.Channel)
}
func (ev *EventHandler) Create(param *Element) (error) {
(...) //Unimportant code
}
func (repo *Repository) CreatePreview(param1 string, param2 string, eventHandler *EventHandler) (*pb.PreviewResponse, error) {
(...)
err := eventHandler.Create(&document)
(...)
preview := <- eventHandler.Channel
(...)
}
func (repo *Repository) SaveAndPublish(param1 string, param2 bool, eventHandler *EventHandler) (*pb.PublishResponse, error) {
(...)
err := eventHandler.Create(&documentToUpdate)
(...)
published := <- eventHandler.Channel
(...)
return repo.SomeOtherMethod(published.ID)
}
Now, on my main.go function, I have the "regular" start of a service, including a gRPC Listener, a HTTP Listener and handling of events. This is done via cmux. So here's a sample of the code (again, code simplified):
func HandlerWrapper(event cloudevents.Event) {
//TODO: HOW DO I HANDLE THIS???
}
// This approach seems to cause issues since it's always the same instance
// var (
// eventHandler = &rep.EventHandler{Channel: make(chan string)}
// )
func (service *RPCService) Start() (err error) {
port, err := strconv.Atoi(os.Getenv("LISTEN_PORT"))
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
// Create multiplexer and listener types
mux := cmux.New(lis)
grpcLis := mux.Match(cmux.HTTP2())
httpLis := mux.Match(cmux.HTTP1())
// *************
// gRPC
// *************
service.server = grpc.NewServer()
reflection.Register(service.server)
pb.RegisterStoryServiceServer(service.server, service)
// *************
// HTTP
// *************
// Declare new CloudEvents Receiver
c, err := kncloudevents.NewDefaultClient(httpLis)
if err != nil {
log.Fatal("Failed to create client, ", err)
}
// *************
// Start Listeners
// *************
// start gRPC server
go func() {
if err := service.server.Serve(grpcLis); err != nil {
log.Fatalf("failed to gRPC serve: %s", err)
}
}()
// start HTTP server
go func() {
// With this line bellow, I'd have to create a Received per eventHandler. Not cool
// log.Fatal(c.StartReceiver(context.Background(), eventHandler.Handle))
// Here we use a wrapper to deal with the event handling and have a single listener
log.Fatal(c.StartReceiver(context.Background(), HandlerWrapper))
}()
if err := mux.Serve(); err != nil {
log.Fatalf("failed to Mux serve: %s", err)
}
return
}
//CreatePreview is used to save a preview for a story
func (service *RPCService) CreatePreview(ctx context.Context, input *pb.PreviewRequest) (*pb.PreviewResponse, error){
eventHandler := &rep.EventHandler{Channel: make(chan string)}
story, err := service.repo.CreatePreview("param1", "param2", eventHandler)
if err != nil {
return nil, err
}
// Return matching the `CreatePreviewResponse` message we created in our
// protobuf definition.
return &pb.PreviewResponse{Story: story}, nil
}
// SaveAndPublish is used to save a story and publish it, returning the story saved.
func (service *RPCService) SaveAndPublish(ctx context.Context, input *pb.PublishRequest) (*pb.PublishResponse, error){
eventHandler := &rep.EventHandler{Channel: make(chan string)}
story, err := service.repo.SaveAndPublish("param1", true, eventHandler)
if err != nil {
return nil, err
}
// Return matching the `SaveAndPublishResponse` message we created in our
// protobuf definition.
return &pb.PublishResponse{Story: story}, nil
}
Now, I know that instead of having to instantiate a single, global eventHandler, in order to use the eventHandler.Handle method on c.StartReceiver() on main.go I can define a wrapper that would, maybe, contain a list of eventHandlers (the HandlerWrapper() method on main.go).
However, I do not know how I could identify which instance of an EventHandler is which and how to properly handle and route these operations, and that is my question:
How do I go about this case where I want to create a Wrapper (a single function to pass into c.StartReceive()) and then let it be handled by the correct instance of Handle()?
I hope the question is clear. I've been trying to get my head around this for a couple days already and can't figure out how to do it.
Presumably, you should be able to differentiate events by using the different sources/methods coming from that event. A quick look at the event spec shows you could split into channels based on source, for example.
The main thing I see that isn't being utilized here is the context object. It seems you could glean the source from that context. This can be seen in their hello world example (check out the receive function).
For your example:
// these are all the handlers for the different sources.
type EventHandlers map[string]CloudEventHandler
var _eventHandlerKey = "cloudEventHandlers"
func HandlerWrapper(ctx context.Context, event cloudevents.Event) {
// Get event source from event context.
src := event.Context.Source
// Then get the appropriate handler for that source (attached to context).
handler := ctx.Value(_eventHandlers).(Values)[src]
// ex: src = "/foo"
handler.SendToChannel(event)
}
func main() {
eventHandlers := make(map[string]CloudEventHandler)
// create all the channels we need, add it to the context.
for _, source := range sourceTypes { // foo bar baz
handler := NewHandler(source)
eventHandlers[source] = handler
}
// start HTTP server
go func() {
// Add the handlers to the context.
context := context.WithValue(context.Background(), _eventHandlerKey, eventHandlers)
log.Fatal(c.StartReceiver(context.Background(), HandlerWrapper))
}
}()
If there are say 3 different sources to be supported, you can use the factory pattern to instantiate those different channels and an interface that all of those implement.
// CloudEventHandler Handles sending cloud events to the proper channel for processing.
type CloudEventHandler interface {
SendToChannel(cloudEvents.Event)
}
type fooHandler struct {channel chan string}
type barHandler struct {channel chan int}
type bazHandler struct {channel chan bool}
func NewHandler(source string) CloudEventHandler {
switch source {
case "/foo":
return &fooHandler{channel: make(chan string, 2)} // buffered channel
case "/bar":
return &barHandler{channel: make(chan int, 2)}
case "/baz":
return &bazHandler{channel: make(chan bool, 2)}
}
}
func (fh *fooHandler) SendToChannel(event CloudEvents.Event) {
var data string
if err := event.DataAs(&data); err != nil {
// (...)
}
go func() {
fh.channel <- data
}()
}
func (bh *barHandler) SendToChannel(event CloudEvents.Event) {
var data int
if err := event.DataAs(&data); err != nil {
// (...)
}
go func() {
bh.channel <- data
}()
}

Go: allow for keyboard input while tcp listener is running

I'm writing an application which should both listen for incoming TCP connections, but also send TCP packets when requested.
I have a simple Listen function, which accepts the incoming connections
func (a *App) Listen() {
for {
conn, err := a.listener.Accept()
if err != nil {
// log error
} else {
go ProcessConn(conn)
}
}
}
I also have a Send function which performs the dialing in order to send a TCP packet
func (a *App) Send(message []byte, host string, port string) error {
conn, err := net.Dial("tcp", host+":"+port)
if err != nil {
return err
} else {
defer conn.Close()
}
_, err = conn.Write(message)
return err
}
Now, when I start my app using the Start function it runs an infinite loop (for the listening for incoming connections), so from the perspective of the terminal the program 'freezes' and does not react to any keyboard operations (expect the CTRL-C).
func (a *App) Start() error {
defer a.Listen()
// perform some unrelated operations
return nil
}
However, I would like to improve my app, in such a way, that it can take users (keyboard) input in order to notify my app that it should send a message (I want the users to be able to give the parameters: message and address of the destination.) I was reading about select statement here: https://gobyexample.com/select. and wanted to use it for my app.
This is what I wrote based on the example from the go website:
func UserReacted(c2 chan string){
for {
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter text: ")
text, _ := reader.ReadString('\n')
c2 <- text
}
}
func main() {
a := NewApp()
a.Start()
c1 := make(chan string)
c2 := make(chan string)
go func() {
UserReacted(c2)
}()
for {
select {
case msg2 := <-c2:
a.Send() // with some parameters added later
default:
}
}
<-finish
}
I understand how the example from the go website works, but I don't know how to use select when one of the function, in my example the Listen function, is not returning any value. I will be very grateful for any advice or example how to do it properly.

Does RPC have a timeout mechanism?

If RPC does not have a timeout mechanism, how do I "kill" an RPC call if it is trying to call an RPC method of a server that is closed?
You can use channels to implement a timeout pattern:
import "time"
c := make(chan error, 1)
go func() { c <- client.Call("Service", args, &result) } ()
select {
case err := <-c:
// use err and result
case <-time.After(timeoutNanoseconds):
// call timed out
}
The select will block until either client.Call returns or timeoutNanoseconds elapsed.
if you want to implement a timeout (to prevent a call from taking too long), then you'll want to change rpc.Dial for net.DialTimeout (notice they're separate packages: rpc vs net). Also be aware that the returned type isn't a client any more (as it is in the previous example); instead it is a 'connection'.
conn, err := net.DialTimeout("tcp", "localhost:8080", time.Minute)
if err != nil {
log.Fatal("dialing:", err)
}
client := rpc.NewClient(conn)
It seems the only solution for net/rpc is to close the underlying connection when you notice stuck requests. Then the client should finish pending requests with "connection broken" errors.
An alternative way is to use https://github.com/valyala/gorpc , which supports timeout RPC calls out of the box.
func (client *Client) Call(serviceMethod string, args interface{}, reply interface{}) error
Call method may block goroutine forever
Change use Go method:
func (client *Client) Go(serviceMethod string, args interface{}, reply interface{}, done chan *Call) *Call
Client example:
call := rpcClient.Go(method, args, reply, make(chan *rpc.Call, 1))
select {
case <-time.After(timeout):
log.Printf("[WARN] rpc call timeout(%v) %v => %v", timeout, rpcClient, s.RpcServer)
rpcClient.Close()
return errors.New("timeout")
case resp := <-call.Done:
if resp != nil && resp.Error != nil {
rpcClient.Close()
return resp.Error
}
Or, anno now, someone might prefer to use context instead. This also takes care of returning a proper error when timed out. (context.DeadlineExceeded)
import (
"context"
"log"
"net/rpc"
)
type Client struct {
*rpc.Client
}
// CallEx is a context aware wrapper around rpc's Client.Call()
func (c *client) CallEx(ctx context.Context, serviceMethod string, args interface{}, reply interface{}) error {
ec := make(chan error, 1)
go func() {
ec <- c.Call(serviceMethod, args, reply)
}()
select {
case err := <-ec:
return err
case <-ctx.Done():
return ctx.Err()
}
}
Invoke this with a Deadlined context:
type Args struct {
A, B int
}
func main(){
rpc, err := rpc.DialHTTP("tcp", "host")
if err != nil {
t.Fatal(err)
}
c := client{rpc}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var i int
if err := c.CallEx(ctx, "Calc.Multiply", Args{2, 2}, &i); err != nil {
log.Fatal(err)
}
}

Resources