How to re-establish grpc bidirectional stream if internet connection is down - go

Question

I am using a go client and server which is connected with grpc bidirectional stream. I need that stream to long running forever without any disconnection, but the stream disconnects within 3 minutes when the internet is down. Is there any way to stop the client from disconnecting or is there any way to reconnect automatically with the server when internet is down. If so please guide me with this. Thankyou.

Answer 1

When gRPC connection is closed, the state of the gRPC client connection will be
IDLE or TRANSIENT_FAILURE. Here is my example for a custom reconnect mechanism for gRPC bi-directional streaming in Go. First, I have a for loop to keep reconnecting until the gRPC server is up, which the state will become ready after calling conn.Connect().
for {
select {
case <-ctx.Done():
return false
default:
if client.Conn.GetState() != connectivity.Ready {
client.Conn.Connect()
}
// reserve a short duration (customizable) for conn to change state from idle to ready if grpc server is up
time.Sleep(500 * time.Millisecond)
if client.Conn.GetState() == connectivity.Ready {
return true
}
// define reconnect time interval (backoff) or/and reconnect attempts here
time.Sleep(2 * time.Second)
}
}
Also, a goroutine will be spawned in order to execute the reconnect tasks. After successfully reconnect, it will spawn another goroutine to listen to gRPC server.
for {
select {
case <-ctx.Done():
return
case <-reconnectCh:
if client.Conn.GetState() != connectivity.Ready && *isConnectedWebSocket {
if o.waitUntilReady(client, isConnectedWebSocket, ctx) {
err := o.generateNewProcessOrderStream(client, ctx)
if err != nil {
logger.Logger.Error("failed to establish stream connection to grpc server ...")
}
// re-listening server side streaming
go o.listenProcessOrderServerSide(client, reconnectCh, ctx, isConnectedWebSocket)
}
}
}
}
Note that the listening task is handled concurrently by another goroutine.
// listening server side streaming
go o.listenProcessOrderServerSide(client, reconnectCh, websocketCtx, isConnectedWebSocket)
You can check out my code example here. Hope this helps.