I've been trying to look through existing posts about closing a channel, but I can't seem to find exactly what I'm looking for. I have two channels inside of a struct. After making the channels, I tried to run defer close(channelName) immediately after, but the channels immediately closed and I got an error. If I don't deal with closing the channels, the code runs fine, but doesn't shut down gracefully and I have to use ctrl z to suspend the program, since it's still running after using ctrl c. Here are the important parts of the code:
type WebsocketClient struct {
pubSocket ws_client.Socket
privSocket ws_client.Socket
pubChan chan interface{}
privChan chan interface{}
}
type KrakenClient struct {
WebSocket WebsocketClient
Testing bool
}
func (client *KrakenClient) initChannels() {
client.WebSocket.pubChan = make(chan interface{})
client.WebSocket.privChan = make(chan interface{})
//defer close(client.WebSocket.pubChan)
//defer close(client.WebSocket.privChan)
}
func (client *KrakenClient) InitWebSocketClient(wg *sync.WaitGroup, testing bool) {
client.initTesting(testing)
client.initChannels()
client.startWebSocketConnection(wg)
}
func (client *KrakenClient) PubDecoder(wg *sync.WaitGroup, ctx context.Context) {
wg.Add(1)
defer wg.Done()
defer client.WebSocket.pubSocket.Close()
if err := PubSocketGuard(client.WebSocket); err != nil { // guard clause checker. makes sure i'm actually using a public WebSocket
panic(err)
}
var res interface{}
ws_client.ReceiveLocker(&client.WebSocket.pubSocket)
client.WebSocket.pubSocket.OnTextMessage = func(message string, socket ws_client.Socket) {
res = ws_client.PubJsonDecoder(message, client.Testing)
client.WebSocket.pubChan <- res
}
ws_client.ReceiveUnlocker(&client.WebSocket.pubSocket)
<-ctx.Done()
log.Println("closing public socket")
return
}
func (client *KrakenClient) PubListen(wg *sync.WaitGroup, ctx context.Context, ohlcMap *types.OHLCVals) {
wg.Add(1)
defer wg.Done()
for c := range client.WebSocket.pubChan {
switch v := c.(type) {
// More code here. Not important to the channel problem
}
<-ctx.Done()
}
func main() {
var testing bool = true
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
kraken := &kraken_client.KrakenClient{}
kraken.InitWebSocketClient(&wg, testing)
go kraken.PubDecoder(&wg, ctx)
kraken.SubscribeToOHLC(&wg, []string{"BTC/USD"}, 5)
ohlcMap := types.OHLCVals{}
go kraken.PubListen(&wg, ctx, &ohlcMap)
<-comms
cancel()
wg.Wait()
}
In summary, struct KrakenClient has a type WebsocketClient, which holds 2 WebSockets and 2 channels. Calling the InitWebSocketClient func on the KrakenClient struct creates the two channels and connects to the WS server. Once connected to the server, I start to unmarshal all responses and send those responses to a channel. I then subscribe to a specific endpoint, and start to "listen" to the responses (actually go through the data and add it to the ohlcMap variable that's passed to the listen func depending on certain circumstances). I just don't understand where I should be closing the channels. Do I need to create a Close function on my KrakenClient struct that defers when the channels are closed? If so, where would it even go? Appreciate any help!
You must call defer in main func because when you call defer in initChannels func it's immediately close your channels after making.
The defer always execute when the function (initChannels in your code) has been return.
you can write a closing function for close the channels and call it on main func like as:
func (client *KrakenClient) closeChannels() {
close(client.WebSocket.pubChan)
close(client.WebSocket.privChan)
}
func (client *KrakenClient) initChannels() {
client.WebSocket.pubChan = make(chan interface{})
client.WebSocket.privChan = make(chan interface{})
}
func main() {
...
kraken.InitWebSocketClient(&wg, testing)
defer kraken.closeChannels()
...
}
Related
I have a websocket client that receives multiple data types. A function unmarshals json received from the server into different structs depending on the data received. The struct is then returned as an interface through a channel to my main file. Since i receive multiple data types from the server, I am not able to specify the exact return value of my parsing function.
With the data in my main file, I would like to have a way to be able to then go through the different values in the data. Since I am returning an interface, this seems impossible to do. Whenever i try to index the interface, I receive an error saying c.VALUE undefined (type interface{} has no field or method VALUE).
I feel like I'm not doing something right here. The 2 solutions I've thought about so far are:
having my channel value be a generic and my listen & JSON decoder funcs (these are all put below) all return a generic or
create an interface with methods. My channel would be of this type and again, my listen & JSON decoder funcs would return this interface.
I'm not sure if either of these ways would actually solve my issue, though. I also don't know if there is one way that would be more performant compared to other ways.
Here is my code to better understand the issue
func main() {
// check if in production or testing mode
var testing bool = true // default to testing
args := os.Args
isTesting(args, &testing, &stored_data.Base_currency)
// go routine handler
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
// set ohlc interval and pairs
OHLCinterval := 5
pairs := []string{"BTC/" + stored_data.Base_currency, "EOS/" + stored_data.Base_currency}
// create ws connections
pubSocket, err := ws_client.ConnectToServer("public", testing)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// create websocket channels
pubCh := make(chan interface{})
defer close(pubCh)
// listen to websocket connections
wg.Add(1)
go pubSocket.PubListen(ctx, &wg, pubCh, testing)
// connect to data streams
pubSocket.SubscribeToOHLC(pairs, OHLCinterval)
// listen to public socket
go func() {
for c := range pubCh {
fmt.Println(c) // This is where I would like to be able to go through my data more thoroughly
}
}()
<-comms
cancel()
wg.Wait()
}
Here is what happens in the PubListen function and my JSON decoding function
func (socket *Socket) PubListen(ctx context.Context, wg *sync.WaitGroup, ch chan interface{}, testing bool) {
defer wg.Done()
defer socket.Close()
var res interface{}
socket.OnTextMessage = func(message string, socket Socket) {
res = pubJsonDecoder(message, testing)
ch <- res
}
<-ctx.Done()
log.Println("closing public socket")
return
}
func pubJsonDecoder(response string, testing bool) interface{} {
var resp interface{}
byteResponse := []byte(response)
resp, err := ohlcResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = heartBeatResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = serverConnectionStatusResponseDecoder(byteResponse, testing)
if err != nil {
resp, err = ohlcSubscriptionResponseDecoder(byteResponse, testing)
}
}
}
return resp
}
Thanks for any help you may have
Since you seem to control the complete list of types which can be unesrialized, you can use a type swicth :
swich v := c.(type) {
case *ohlcResponse:
// in this block, v is a *ohlcRrsponse
case *heartBeatResponse:
// in this block, v is a *heartBeatResponse
case *serverConnectionStatusResponse:
// in this block, v is a *serverConnectionStatus
case *ohlcSubscriptionResponse:
// in this block, v is a *ohlcSubscriptionResponse
default:
// choose some way to report unhandled types:
log.Fatalf("unhandled response type: %T", c)
}
I'm running a websocket client and want to pass the response(s) from the client to a channel that i can work with in my main file. Currently, the channel just returns a nil value once and then nothing else. I seem to have an issue when passing a value to the channel. Any help? Here is what I've done so far
package main
import (
"context"
"fmt"
"kraken_client/stored_data"
"kraken_client/ws_client"
"os"
"os/signal"
"strings"
"sync"
"syscall"
)
func main() {
// check if in production or testing mode & find base curency
var testing bool = true
args := os.Args
isTesting(args, &testing, &stored_data.Base_currency)
// go routine handler
comms := make(chan os.Signal, 1)
signal.Notify(comms, os.Interrupt, syscall.SIGTERM)
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
var wg sync.WaitGroup
// set ohlc interval and pairs
OHLCinterval := 5
pairs := []string{"BTC/" + stored_data.Base_currency, "EOS/" + stored_data.Base_currency}
// create ws connections
pubSocket, err := ws_client.ConnectToServer("public", testing)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// listen to websocket connections
ch := make(chan interface{})
wg.Add(1)
go pubSocket.PubListen(ctx, &wg, ch, testing)
// subscribe to a stream
pubSocket.SubscribeToOHLC(pairs, OHLCinterval)
go func() {
for c := range ch {
fmt.Println(c)
}
}()
<-comms
cancel()
wg.Wait()
defer close(ch)
}
Here is how the PubListen function works
func (socket *Socket) PubListen(ctx context.Context, wg *sync.WaitGroup, ch chan interface{}, testing bool) {
defer wg.Done()
defer socket.Close()
var res interface{}
socket.OnTextMessage = func(message string, socket Socket) {
//log.Println(message)
res = pubJsonDecoder(message, testing) // this function decodes the message and returns an interface
log.Println(res) // this is printing the correctly decoded value.
}
ch <- res
log.Println(res) // does not print a value
log.Println(ch) // does not print a value
<-ctx.Done()
log.Println("closing public socket")
return
}
What am I doing wrong?
The code in the question executes the statement ch <- res once from PubListen before the res is set by the OnTextMessage function.
To send a value to ch on each message, move the line ch <- res to the OnTextMessage function. That function is called once for each message.
func (socket *Socket) PubListen(ctx context.Context, wg *sync.WaitGroup, ch chan interface{}, testing bool) {
defer wg.Done()
defer socket.Close()
socket.OnTextMessage = func(message string, socket Socket) {
res := pubJsonDecoder(message, testing)
ch <- res
log.Println(res)
}
<-ctx.Done()
log.Println("closing public socket")
return
}
I am trying to build a pipeline for ingesting data with Go. The 3 stages are "Download batches", "Transform each message", and "Enqueue messages into a queue".
The logic which seemed natural to me is to create 3 functions for each stage, and to tie these functions with unbuffered channels.
Somewhere in my code, I am not implementing channels correctly, or not using waitgroups? As only 1 message gets to the final stage and program seems to stop/block.
func (c *worker) startWork() {
// channel for messages to be sent to the queue
chMessagesToEnqueue := make(chan types.Foo)
// channel for messages to be transformed
chMessagesToTransform := make(chan []types.UpstreamFooType)
// start the goroutines with the channels
go c.startTransformer(chMessagesToTransform, chMessagesToEnqueue)
go c.startEnqueuer(chMessagesToEnqueue)
go c.startDownloader(chMessagesToTransform)
}
func (c *worker) startDownloader(out chan []types.UpstreamFooType) {
// https://github.com/SebastiaanKlippert/go-soda
// uses a library here to fetch data from upstream APIs, but the gist is:
var wg sync.WaitGroup
for i := 0; i < c.workerCount; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for {
// i've cut out some meat out to make more concise
var results []types.UpstreamFooType
err = json.NewDecoder(resp.Body).Decode(&results)
out <- results
}
}()
}
wg.Wait()
}
func (c *worker) startTransformer(in <-chan []types.UpstreamFooType, out chan types.Foo) {
data := <-in
for _, record := range data {
msg := types.Foo{
name: record.fifa,
}
out <- msg
}
}
func (c *worker) startEnqueuer(in <-chan []types.Foo) {
data := <-in
c.logger.Infow("startEnqueuer", "data", data)
}
I have a go function processing that use two distinct goroutines. produce will push some data into a channel and consume will read these data. Here is an example:
type MyObject struct{
...
}
func processing() {
var wg sync.WaitGroup
dataChannel := make(chan MyObject, 5)
wg.Add(2)
go produce(wg, dataChannel)
go consume(wg, dataChannel)
wg.Wait()
}
func produce (wg *sync.WaitGroup, dataChannel chan MyObject){
for{
// Produce data in dataChannel
}
}
func consume (wg *sync.WaitGroup, dataChannel chan MyObject){
for{
// Consume data from dataChannel
}
}
I want my processing function to be started and stoped by an HTTP call. So I am looking to do something as follow:
func main() {
// echo instance
e := echo.New()
e.GET("/", startProcessing)
e.Logger.Fatal(e.Start(":8099"))
}
func startProcessing(c echo.Context) error{
command := c.QueryParam("command")
if(command == "start"){
processing()
}else if(command == "stop"){
if (/* ? processing is running ? */){
/* ? stop processing process? */
}
}
}
What is the correct way to do this with Go?
Here how to start and stop a function using context, try this:
package main
import (
"context"
"fmt"
"sync"
"time"
)
func main() {
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
dataChannel := make(chan MyObject, 5)
wg.Add(2)
go produce(ctx, &wg, dataChannel)
go consume(&wg, dataChannel)
time.Sleep(1 * time.Second)
cancel() // cancel when we are finished consuming data
wg.Wait()
}
func produce(ctx context.Context, wg *sync.WaitGroup, dataChannel chan MyObject) {
defer wg.Done()
i := 1
for {
select {
case <-ctx.Done():
close(dataChannel)
return // returning not to leak the goroutine
case dataChannel <- MyObject{i}:
i++
time.Sleep(250 * time.Millisecond)
}
}
}
func consume(wg *sync.WaitGroup, dataChannel chan MyObject) {
defer wg.Done()
for v := range dataChannel {
fmt.Println(v)
}
}
type MyObject struct {
i int
}
For HTTP you need to do it yourself!
It needs to have some concurrent safe ID or map or something to keep track of how many functions you called and then call a cancel() to stop it.
I work the first time with goroutines and channels in go and do not come any further.
I have a websocket connection where every time a user connect a new goroutine is spawned. Now I want to stop this goroutine if the user disconnect from the websocket connection.
To manage the stop signale I have create a map of channels. Each entry can be identified by the users websocket connection. I pass the websocket connection, the map of channels for the stop signal and two other parameters to the goroutine. But the goroutine doesn't receive any values from the quit channel and I don't know why.
Here is the relevant code for the main package:
package main
import (
"net/http"
"time"
"github.com/gorilla/websocket"
)
func wsHandler(w http.ResponseWriter, r *http.Request) {
...
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
return
}
defer ws.Close()
data.Quit[ws] = make(chan bool)
data.DB.ListenToTable(data.GetTableName(source), channel, data.Quit, ws)
for {
if _, _, err := ws.NextReader(); err != nil {
data.Quit[ws] <- true
ws.Close()
break
}
}
}
And the code of the data package where the goroutine is created:
package data
var Quit = make(map[*websocket.Conn](chan bool))
func (db *rethinkDB) ListenToTable(name string, ch chan Data, quit map[*websocket.Conn](chan bool), ws *websocket.Conn) {
go func(name string, ws *websocket.Conn) {
for {
select {
case <-quit[ws]:
fmt.Println("QUIT GOROUTINE")
break
default:
res, err := r.Table(name).Changes().Run(db.session)
if err != nil {
log.Fatalln(err)
}
var response DataFeed
for res.Next(&response) {
response.NewVal.WS = ws
ch <- response.NewVal
}
if res.Err() != nil {
log.Println(res.Err())
}
}
}
}(name, ws)
}
I have also tried buffered channels or pass the channel instead of the map of channels to the goroutine but without success. The fmt.Println("QUIT GOROUTINE") command is never called and the goroutine isn't sopped.
I hope someone can help me and sorry if this question was already ask but I haven't found a solution that solves my problem.
First to make things easier:
As far as I can see you do not need a global register for the quit channels. Just create a ch := make(chan bool) in main, pass it to ListenToTable (instead of the whole map of channels) and use it in the select. In main close(ch) it if you want to exit. But as you said, that doesn't solve your problem.
Theoretically you are on the right track with closing the go routine. I took your sample code and made the following runnable code from it:
package main
import (
"fmt"
"time"
)
func main() {
chClose := make(chan bool)
channel := make(chan string)
ListenToTable("somestring", channel, chClose)
time.Sleep(3 * time.Second)
chClose <- true
time.Sleep(1 * time.Second)
}
func ListenToTable(name string, ch chan string, chClose chan bool) {
go func(name string) {
for {
select {
case <-chClose:
fmt.Println("QUIT GOROUTINE")
return // VERY IMPORTANT: not break!
default:
}
}
}(name)
}
The problem must be with something else in you code, probably blocked by something in the default section and not even executing the select. Try printing fmt.Println("something") before the select {. If that is not printed regularly then you have your answer.
One more thing: As commented in the code above you cannot break out of a for { select { ... } } with a single break. You need to use a return (to exit the function) or another tactic (like a break with a label as Adrian suggested in the comments). The break will only exit the select, but not the for loop.