How to create server for persistent stream (aka pubsub) in Golang GRPC - go

Question

I am building service that needs to send events to all subscribed consumers in Pub/Sub manner eg. send one event to all currently connected clients.
I am using Protobuf for that with the following proto definition:
service EventsService {
rpc ListenForEvents (AgentProcess) returns (stream Event) {}
}
Both server & client are written in Go.
My problem is that when client initiates connection then the stream it is not long-lived, eg. when server returns from ListenForEvents method:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
//persist listener here so it can be used later when backend needs to send some messages to client
return nil
}
then the client almost instantly gets EOF error which means that server probably closed connection.
What do I do so that the client is subscribed for a long time to the server? The main problem is that I might not have anything to send to the client when it calls ListenForEvents method on the server, this is why I want this stream to be long lived to be able to send messages later.

Answer 1

The stream terminates when you return from the server function. Instead, you should receive events somehow, and send them to the client without returning from your server. There are probably many ways you can do this. Below is the sketch of one way of doing it.
This relies on the server connection running on a separate goroutine. There is a Broadcast() function that will send messages to all connected clients. It looks like this:
var allRegisteredClients map[*pb.AgentProcess]chan Message
var clientsLock sync.RWMutex{}
func Broadcast(msg Message) {
clientsLock.RLock()
for _,x:=range allRegisteredClients {
x<-msg
}
clientsLock.RUnlock()
}
Then, your clients have to register themselves, and process messages:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
clientsLock.Lock()
ch:=make(chan Message)
allRegisteredClients[process]=ch
clientsLock.Unlock()
for msg:=range ch {
// send message
// Deal with errors
// Deal with client terminations
}
clientsLock.Lock()
delete(allRegisteredClients,process)
clientsLock.Unlock()
}
As I said, this is only a sketch of the idea.

Answer 2

I have managed to nail it down.
Basically I never return from method ListenForEvents.
It creates channel, persists in global-like map of subscribed clients and keeps reading from that channel indefinitely.
The whole implementation of server logic:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
chans, exists := e.listeners[process.Hostname]
chanForThisClient := make(chan *pb.Event)
if !exists {
e.listeners[process.Hostname] = []chan *pb.Event{chanForThisClient}
} else {
e.listeners[process.Hostname] = append(chans, chanForThisClient)
}
for {
select {
case <-listener.Context().Done():
return nil
case res := <-chanForThisClient:
_ = listener.Send(res)
}
}
return nil
}

Answer 3

You need to provide keepalive settings for grpc client and server
See details here https://github.com/grpc/grpc/blob/master/doc/keepalive.md
Examples https://github.com/grpc/grpc-go/tree/master/examples/features/keepalive