Pulling 0-sized golang chan - go

My use-case is the following: I need to send POST requests to 0...N subscribers, which are represented by a targetUrl. I want to limitate the max number of goroutine to let's say, 100. My code (simplified) is the following:
package main
import (
"fmt"
"log"
"net/http"
"errors"
)
const MAX_CONCURRENT_NOTIFICATIONS = 100
type Subscription struct {
TargetUrl string
}
func notifySubscribers(subs []Subscription) {
log.Println("notifySubscribers")
var buffer = make(chan Subscription, len(subs))
defer close(buffer)
for i := 0; i < MAX_CONCURRENT_NOTIFICATIONS; i++ {
go notifySubscriber(buffer)
}
for i := range subs {
buffer <- subs[i]
}
}
func notifySubscriber(buffer chan Subscription) {
log.Println("notifySubscriber")
for {
select {
case sub := <-buffer:
log.Println("sending notification to " + sub.TargetUrl)
resp, err := failPost()
if err != nil {
log.Println(fmt.Sprintf("failed to notify %s. error: %s", sub.TargetUrl, err.Error()))
} else {
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Println(fmt.Sprintf("%s responded with %d", sub.TargetUrl, resp.StatusCode))
}
}
}
log.Println(fmt.Sprintf("buffer size: %d", len(buffer)))
}
}
func failPost() (*http.Response, error) {
return &http.Response{
StatusCode: http.StatusBadRequest,
}, errors.New("some bad error")
}
func main() {
log.Println("main")
var subs []Subscription
subs = append(subs, Subscription{TargetUrl: "http://foo.bar"})
subs = append(subs, Subscription{TargetUrl: "http://fizz.buzz"})
notifySubscribers(subs)
select {}
}
The output is the following:
2018/01/24 10:52:48 failed to notify . error: some bad error
2018/01/24 10:52:48 buffer size: 1
2018/01/24 10:52:48 sending notification to
2018/01/24 10:52:48 failed to notify . error: some bad error
2018/01/24 10:52:48 buffer size: 0
2018/01/24 10:52:48 sending notification to
2018/01/24 10:52:48 failed to notify . error: some bad error
... and so on till I SIGINT the program
So basically it means that I've successfuly send the notifications to the right people, but I still continue to send to empty targetUrl because I read from an empty chan.
What is wrong ?
[EDIT] Workaround, but I don't like it
for {
select {
case sub, more := <-buffer:
if !more {
return
}
}
}

It's because you are closing the buffer but your notifySubscriber is still listening on the buffer. A closed channel always returns the default type value(in this case an empty Subscription with empty TargetURL). Hence, you are getting an empty string.
Scenarios:
If you want to keep the goroutines running, then don't close the buffer.
Stop the goroutines once the work is done and then close the buffer.

From the spec:
For a channel c, the built-in function close(c) records that no more
values will be sent on the channel. It is an error if c is a
receive-only channel. Sending to or closing a closed channel causes a
run-time panic. Closing the nil channel also causes a run-time panic.
After calling close, and after any previously sent values have been
received, receive operations will return the zero value for the
channel's type without blocking. The multi-valued receive operation
returns a received value along with an indication of whether the
channel is closed.
The last sentence means that sub, more := <-buffer, more will be false if buffer is closed.
However, in your case, the code can use some improvement.
First, it makes no sense to use a select statement where there are only one case. It would just act the same without the select.
Second, in cases that the recieving channel is guaranteed to return, range over channel can be used. So your code can be changed to:
func notifySubscriber(buffer chan Subscription) {
log.Println("notifySubscriber")
for sub:= range buffer {
//Code here...
}
}

Related

Golang unexpected behaviour while implementing concurrency using channel and select statement

Below is my implementation of the Pub/Sub model in golang using concurrency. The code executes as expected sometimes but sometimes it gives output like this:
Output
message received on channel 2: Hello World
message received on channel 3: Hello World
message received on channel 1: Hello World
message received on channel 1:
subscriber 1's context cancelled
message received on channel 3: Only channels 2 and 3 should print this
message received on channel 2: Only channels 2 and 3 should print this
Key thing to note here is that subscriber 1 printed message received on channel 1: even after there is a call in the main function to remove it as a subscriber. Subscriber 1 should only receive one message. What seems to be happening is that subscriber receives a message from subscriber.out before the dctx is cancelled. How can I fix this. The subscriber's goroutine is launched in the AddSubscriber method.
Expected execution:
Channel 1 should only print once and then it's context should be cancelled and it's goroutine should exit.
main.go
package main
import (
"fmt"
"net/http"
)
func main() {
var err error
publisher := NewPublisher()
publisher.AddSubscriber()
publisher.AddSubscriber()
publisher.AddSubscriber()
publisher.Start()
err = publisher.Publish("Hello World")
if err != nil {
fmt.Printf("could not publish: %v\n", err)
}
err = publisher.RemoveSubscriber(1)
if err != nil {
fmt.Printf("could not remove subscriber: %v\n", err)
}
err = publisher.Publish("Only channels 2 and 3 should print this")
if err != nil {
fmt.Printf("could not publish: %v\n", err)
}
// this is merely to keep the server running
http.ListenAndServe(":8080", nil)
}
publisher.go
package main
import (
"context"
"errors"
"fmt"
"sync"
)
// Publisher sends messages received in its in channel
// to all the Subscribers listening to it.
type Publisher struct {
sequence uint
in chan string
subscribers map[uint]*Subscriber
sync.RWMutex
ctx context.Context
cancel *context.CancelFunc
}
// NewPublisher returns a new Publisher with zero subscribers.
// A Publishers needs to be started with the Publisher.Start()
// method before it can start publishing incoming messages.
func NewPublisher() *Publisher {
ctx, cancel := context.WithCancel(context.Background())
return &Publisher{subscribers: map[uint]*Subscriber{}, ctx: ctx, cancel: &cancel}
}
// AddSubscriber creates a new Subscriber that starts
// listening to the messages sent by this Publisher.
func (p *Publisher) AddSubscriber() {
dctx, cancel := context.WithCancel(p.ctx)
p.Lock()
nextId := p.sequence + 1
subscriber := NewSubscriber(nextId, dctx, &cancel)
p.subscribers[nextId] = subscriber
p.sequence = p.sequence + 1
p.Unlock()
go func() {
for {
select {
case <-p.ctx.Done():
fmt.Printf("parent context cancelled\n")
(*subscriber.cancel)()
return
case <-dctx.Done():
fmt.Printf("subscriber %d's context cancelled\n", subscriber.id)
return
case msg := <-subscriber.out:
fmt.Printf("message received on channel %d: %s\n", subscriber.id, msg)
}
}
}()
}
// Publish sends the msg to all the subscribers subscribed to it.
func (p *Publisher) Publish(msg string) error {
// publish only if Publisher has been started
if p.in == nil {
return errors.New("publisher not started yet")
}
// publish only if subscriber count > 0
p.RLock()
if len(p.subscribers) == 0 {
return errors.New("no subscribers to receive the message")
}
p.RUnlock()
// send message to in channel
// establish lock to ensure no modifications take place
// while the message is being sent
p.Lock()
p.in <- msg
p.Unlock()
return nil
}
// Start initializes the the publishers in channel
// and makes it ready to start publishing incoming messages.
func (p *Publisher) Start() {
in := make(chan string)
p.in = in
go func() {
for {
select {
case <-p.ctx.Done():
// using break here only breaks out of select statement
// instead of breaking out of the loop
fmt.Printf("done called on publisher\n")
return
case msg := <-p.in:
p.RLock()
for _, subscriber := range p.subscribers {
subscriber.out <- msg
}
p.RUnlock()
}
}
}()
}
// Stop prevents the Publisher from listening to any
// incoming messages by closing the in channel.
func (p *Publisher) Stop() {
// should I also remove all subscribers to prevent it from panicking?
(*p.cancel)()
}
// RemoveSubscriber removes the subscriber specified by the given id.
// It returns an error "could not find subscriber"
// if Subscriber with the given id is not subscribed to the Publisher.
func (p *Publisher) RemoveSubscriber(id uint) error {
p.Lock()
defer p.Unlock()
subscriber, ok := p.subscribers[id]
if !ok {
return errors.New("could not find subscriber")
}
(*subscriber.cancel)()
delete(p.subscribers, id)
close(subscriber.out)
return nil
}

read tcp read: connection reset by peer

I've been using the Golang DynamoDB SDK for a while now, and recently I started seeing this error type come back:
RequestError: send request failed
caused by: Post "https://dynamodb.[REGION].amazonaws.com/": read tcp [My IP]->[AWS IP]: read: connection reset by peer
This only seems to occur when writing large amounts of data to DynamoDB, although the error is not limited to any particular type of request. I've seen it in both UpdateItem and BatchWriteItem requests. Furthermore, as the failure isn't consistent, I can't localize it to a particular line of code. It seems that the error is related to some sort of network issue between my service and AWS but, as it doesn't come back as a throttling exception, I'm not sure how to debug it. Finally, as the response comes back from a write request, I don't think retry logic is really the solution here either.
Here's my batch-write code:
func (conn *Connection) BatchWrite(tableName string, requests []*dynamodb.WriteRequest) error {
// Get the length of the requests; if there aren't any then return because there's nothing to do
length := len(requests)
log.Printf("Attempting to write %d items to DynamoDB", length)
if length == 0 {
return nil
}
// Get the number of requests to make
numRequests := length / 25
if length%25 != 0 {
numRequests++
}
// Create the variables necessary to manage the concurrency
var wg sync.WaitGroup
errs := make(chan error, numRequests)
// Attempt to batch-write the requests to DynamoDB; because DynamoDB limits the number of concurrent
// items in a batch request to 25, we'll chunk the requests into 25-report segments
sections := make([][]*dynamodb.WriteRequest, numRequests)
for i := 0; i < numRequests; i++ {
// Get the end index which is 25 greater than the current index or the end of the array
// if we're getting close
end := (i + 1) * 25
if end > length {
end = length
}
// Add to the wait group so that we can ensure all the concurrent processes finish
// before we close down the process
wg.Add(1)
// Write the chunk to DynamoDB concurrently
go func(wg *sync.WaitGroup, index int, start int, end int) {
defer wg.Done()
// Call the DynamoDB operation; record any errors that occur
if section, err := conn.batchWriteInner(tableName, requests[start:end]); err != nil {
errs <- err
} else {
sections[index] = section
}
}(&wg, i, i*25, end)
}
// Wait for all the goroutines to finish
wg.Wait()
// Attempt to read an error from the channel; if we get one then return it
// Otherwise, continue. We have to use the select here because this is
// the only way to attempt to read from a channel without it blocking
select {
case err, ok := <-errs:
if ok {
return err
}
default:
break
}
// Now, we've probably gotten retries back so take these and combine them into
// a single list of requests
retries := sections[0]
if len(sections) > 1 {
for _, section := range sections[1:] {
retries = append(retries, section...)
}
}
// Rewrite the requests and return the result
return conn.BatchWrite(tableName, retries)
}
func (conn *Connection) batchWriteInner(tableName string, requests []*dynamodb.WriteRequest) ([]*dynamodb.WriteRequest, error) {
// Create the request
request := dynamodb.BatchWriteItemInput{
ReturnConsumedCapacity: aws.String(dynamodb.ReturnConsumedCapacityNone),
ReturnItemCollectionMetrics: aws.String(dynamodb.ReturnItemCollectionMetricsNone),
RequestItems: map[string][]*dynamodb.WriteRequest{
tableName: requests,
},
}
// Attempt to batch-write the items with an exponential backoff
var result *dynamodb.BatchWriteItemOutput
err := backoff.Retry(func() error {
// Attempt the batch-write; if it fails then back-off and wait. Otherwise break out
// of the loop and return
var err error
if result, err = conn.inner.BatchWriteItem(&request); err != nil {
// If we have an error then what we do here will depend on the error code
// If the error code is for exceeded throughput, exceeded request limit or
// an internal server error then we'll try again. Otherwise, we'll break out
// because the error isn't recoverable
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case dynamodb.ErrCodeProvisionedThroughputExceededException:
case dynamodb.ErrCodeRequestLimitExceeded:
case dynamodb.ErrCodeInternalServerError:
return err
}
}
// We received an error that won't be fixed by backing off; return this as a permanent
// error so we can tell the backoff library that we want to break out of the exponential backoff
return backoff.Permanent(err)
}
return nil
}, backoff.NewExponentialBackOff())
// If the batch-write failed then return an error
if err != nil {
return nil, err
}
// Roll the unprocessed items into a single list and return them
var list []*dynamodb.WriteRequest
for _, item := range result.UnprocessedItems {
list = append(list, item...)
}
return list, nil
}
Has anyone else dealt with this issue before? What's the correct approach here?

Syncing websocket loops with channels in Golang

I'm facing a dilemma here trying to keep certain websockets in sync for a given user. Here's the basic setup:
type msg struct {
Key string
Value string
}
type connStruct struct {
//...
ConnRoutineChans []*chan string
LoggedIn bool
Login string
//...
Sockets []*websocket.Conn
}
var (
//...
/* LIST OF CONNECTED USERS AN THEIR IP ADDRESSES */
guestMap sync.Map
)
func main() {
post("Started...")
rand.Seed(time.Now().UTC().UnixNano())
http.HandleFunc("/wss", wsHandler)
panic(http.ListenAndServeTLS("...", "...", "...", nil))
}
func wsHandler(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Origin")+":8080" != "https://...:8080" {
http.Error(w, "Origin not allowed", 403)
fmt.Println("Client origin not allowed! (https://"+r.Host+")")
fmt.Println("r.Header Origin: "+r.Header.Get("Origin"))
return
}
///
conn, err := websocket.Upgrade(w, r, w.Header(), 1024, 1024)
if err != nil {
http.Error(w, "Could not open websocket connection", http.StatusBadRequest)
fmt.Println("Could not open websocket connection with client!")
}
//ADD CONNECTION TO guestMap IF CONNECTION IS nil
var authString string = /*gets device identity*/;
var authChan chan string = make(chan string);
authValue, authOK := guestMap.Load(authString);
if !authOK {
// NO SESSION, CREATE A NEW ONE
newSession = getSession();
//defer newSession.Close();
guestMap.Store(authString, connStruct{ LoggedIn: false,
ConnRoutineChans: []*chan string{&authChan},
Login: "",
Sockets: []*websocket.Conn{conn}
/* .... */ });
}else{
//SESSION STARTED, ADD NEW SOCKET TO Sockets
var tempConn connStruct = authValue.(connStruct);
tempConn.Sockets = append(tempConn.Sockets, conn);
tempConn.ConnRoutineChans = append(tempConn.ConnRoutineChans, &authChan)
guestMap.Store(authString, tempConn);
}
//
go echo(conn, authString, &authChan);
}
func echo(conn *websocket.Conn, authString string, authChan *chan string) {
var message msg;
//TEST CHANNEL
authValue, _ := guestMap.Load(authString);
go sendToChans(authValue.(connStruct).ConnRoutineChans, "sup dude?")
fmt.Println("got past send...");
for true {
select {
case val := <-*authChan:
// use value of channel
fmt.Println("AuthChan for user #"+strconv.Itoa(myConnNumb)+" spat out: ", val)
default:
// if channels are empty, this is executed
}
readError := conn.ReadJSON(&message)
fmt.Println("got past readJson...");
if readError != nil || message.Key == "" {
//DISCONNECT USER
//.....
return
}
//
_key, _value := chief(message.Key, message.Value, &*conn, browserAndOS, authString)
if writeError := conn.WriteJSON(_key + "|" + _value); writeError != nil {
//...
return
}
fmt.Println("got past writeJson...");
}
}
func sendToChans(chans []*chan string, message string){
for i := 0; i < len(chans); i++ {
*chans[i] <- message
}
}
I know, a big block of code eh? And I commented out most of it...
Anyway, if you've ever used a websocket most of it should be quite familiar:
1) func wsHandler() fires every time a user connects. It makes an entry in guestMap (for each unique device that connects) which holds a connStruct which holds a list of channels: ConnRoutineChans []*chan string. This all gets passed to:
2) echo(), which is a goroutine that constantly runs for each websocket connection. Here I'm just testing out sending a message to other running goroutines, but it seems my for loop isn't actually constantly firing. It only fires when the websocket receives a message from the open tab/window it's connected to. (If anyone can clarify this mechanic, I'd love to know why it's not looping constantly?)
3) For each window or tab that the user has open on a given device there is a websocket and channel stored in an arrays. I want to be able to send a message to all the channels in the array (essentially the other goroutines for open tabs/windows on that device) and receive the message in the other goroutines to change some variables set in the constantly running goroutine.
What I have right now works only for the very first connection on a device, and (of course) it sends "sup dude?" to itself since it's the only channel in the array at the time. Then if you open a new tab (or even many), the message doesn't get sent to anyone at all! Strange?... Then when I close all the tabs (and my commented out logic removes the device item from guestMap) and start up a new device session, still only the first connection gets it's own message.
I already have a method for sending a message to all the other websockets on a device, but sending to a goroutine seems to be a little more tricky than I thought.
To answer my own question:
First, I've switched from a sync.map to a normal map. Secondly, in order for nobody to be reading/writing to it at the same time I've made a channel that you call to do any read/write operation on the map. I've been trying my best to keep my data access and manipulation quick to execute so the channel doesn't get crowded so easily. Here's a small example of that:
package main
import (
"fmt"
)
var (
guestMap map[string]*guestStruct = make(map[string]*guestStruct);
guestMapActionChan = make (chan actionStruct);
)
type actionStruct struct {
Action func([]interface{})[]interface{}
Params []interface{}
ReturnChan chan []interface{}
}
type guestStruct struct {
Name string
Numb int
}
func main(){
//make chan listener
go guestMapActionChanListener(guestMapActionChan)
//some guest logs in...
newGuest := guestStruct{Name: "Larry Josher", Numb: 1337}
//add to the map
addRetChan := make(chan []interface{})
guestMapActionChan <- actionStruct{Action: guestMapAdd,
Params: []interface{}{&newGuest},
ReturnChan: addRetChan}
addReturned := <-addRetChan
fmt.Println(addReturned)
fmt.Println("Also, numb was changed by listener to:", newGuest.Numb)
// Same kind of thing for removing, except (of course) there's
// a lot more logic to a real-life application.
}
func guestMapActionChanListener (c chan actionStruct){
for{
value := <-c;
//
returned := value.Action(value.Params);
value.ReturnChan <- returned;
close(value.ReturnChan)
}
}
func guestMapAdd(params []interface{}) []interface{} {
//.. do some parameter verification checks
theStruct := params[0].(*guestStruct)
name := theStruct.Name
theStruct.Numb = 75
guestMap[name] = &*theStruct
return []interface{}{"Added '"+name+"' to the guestMap"}
}
For communication between connections, I just have each socket loop hold onto their guestStruct, and have more guestMapActionChan functions that take care of distributing data to other guests' guestStructs
Now, I'm not going to mark this as the correct answer unless I get some better suggestions as how to do something like this the right way. But for now this is working and should guarantee no races for reading/writing to the map.
Edit: The correct approach should really have been to just use a sync.Mutex like I do in the (mostly) finished project GopherGameServer

Closing a channel from the receiver side: deadlock when accessing sync.Mutex from multiple goroutines

I'm trying to implement graceful channel closing from receiver side.
Yes, I'm aware that this violates the channel closing rule:
...don't close a channel from the receiver side and don't close a channel if the channel has multiple concurrent senders.
But I want to implement such logic. Unfortunately, I fail into deadlock issue in the number of cases: the application just hangs for the unlimited time, trying to lock same locked Mutex again.
So, I have 2 goroutines:
one that will write into a channel and
another that will receive data + will close channel from the receiver side.
My channel wrapped in the struct with sync.Mutex and closed boolean flag:
type Chan struct {
sync.Mutex // can be replaced with deadlock.Mutex from "github.com/sasha-s/go-deadlock"
data chan int
closed bool
}
All Send(), Close(), IsClosed() operations on this struct are guarded with Mutex and to prevent duplicate locking have the non-threadsafe method versions (send(), close(), isClosed()).
The full source code:
package main
import (
"log"
"net/http"
"sync"
)
func main() {
log.Println("Start")
ch := New(0) // unbuffered channel to expose problem faster
wg := sync.WaitGroup{}
wg.Add(2)
// send data:
go func(ch *Chan) {
for i := 0; i < 100; i++ {
ch.Send(i)
}
wg.Done()
}(ch)
// receive data and close from receiver side:
go func(ch *Chan) {
for data := range ch.data {
log.Printf("Received %d data", data)
// Bad practice: I want to close the channel from receiver's side:
if data > 50 {
ch.Close()
break
}
}
wg.Done()
}(ch)
wg.Wait()
log.Println("End")
}
type Chan struct {
deadlock.Mutex //sync.Mutex
data chan int
closed bool
}
func New(size int) *Chan {
defer func() {
log.Printf("Channel was created")
}()
return &Chan{
data: make(chan int, size),
}
}
func (c *Chan) Send(data int) {
c.Lock()
c.send(data)
c.Unlock()
}
func (c *Chan) Close() {
c.Lock()
c.close()
c.Unlock()
}
func (c *Chan) IsClosed() bool {
c.Lock()
defer c.Unlock()
return c.isClosed()
}
// send is internal non-threadsafe api.
func (c *Chan) send(data int) {
if !c.closed {
c.data <- data
log.Printf("Data %d was sent", data)
}
}
// close is internal non-threadsafe api.
func (c *Chan) close() {
if !c.closed {
close(c.data)
c.closed = true
log.Println("Channel was closed")
} else {
log.Println("Channel was already closed")
}
}
// isClosed is internal non-threadsafe api.
func (c *Chan) isClosed() bool {
return c.closed
}
You can run this program in the sandbox.
On the local machine, in small number of runs, after 30 seconds the output will be (using deadlock.Mutex instead of sync.Mutex):
2018/04/01 11:26:22 Data 50 was sent
2018/04/01 11:26:22 Received 50 data
2018/04/01 11:26:22 Data 51 was sent
2018/04/01 11:26:22 Received 51 data
POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 35 lock 0xc42015a040
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:71 main.(*Chan).Send { c.Lock() } <<<<<
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:30 main.main.func1 { ch.Send(i) }
Have been trying to lock it again for more than 30s
goroutine 36 lock 0xc42015a040
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:77 main.(*Chan).Close { c.Lock() } <<<<<
close-from-receiver-side/closeFromReceiverSideIsBadPractice.go:44 main.main.func2 { ch.Close() }
Why this deadlock happened and how to fix this implementation to avoid deadlocks?
Closing the channel on the sender's side is not the answer. So, this is not the fix for my question: Example of closing channel from sender side.
Send grabs the lock, then attempts to sends data down the channel. This may happen just after the 50th receive operation. There will be no more receives, so c.data <- data blocks forever and consequently the Mutex is held forever.
For cancellation, use another channel (instead of the boolean) and a select statement in Send. You can also leverage the context package.
You can try as hard as you like: you have to close the channel from sender side.
You might be able to get it working without a complete lockdown but you will leak goroutines. The sender will block forever and cannot be shut down. If the receiver wants to trigger a shutdown it has to tell the sender to shut the channel down. How you could tell the sender to shut down:
A boolean as you suggest (needs another mutex)
A stop-channel that when closed signals the sender to close the data channel (cannot be closed multiple times)
a ctx.Context: calling the cancel() function will signal the sender to stop. (Can be cancelled multiple times without worry)
(Only elaborating on Peters correct answer)

Graceful shutdown of gRPC downstream

Using the following proto buffer code :
syntax = "proto3";
package pb;
message SimpleRequest {
int64 number = 1;
}
message SimpleResponse {
int64 doubled = 1;
}
// All the calls in this serivce preform the action of doubling a number.
// The streams will continuously send the next double, eg. 1, 2, 4, 8, 16.
service Test {
// This RPC streams from the server only.
rpc Downstream(SimpleRequest) returns (stream SimpleResponse);
}
I'm able to successfully open a stream, and continuously get the next doubled number from the server.
My go code for running this looks like :
ctxDownstream, cancel := context.WithCancel(ctx)
downstream, err := testClient.Downstream(ctxDownstream, &pb.SimpleRequest{Number: 1})
for {
responseDownstream, err := downstream.Recv()
if err != io.EOF {
println(fmt.Sprintf("downstream response: %d, error: %v", responseDownstream.Doubled, err))
if responseDownstream.Doubled >= 32 {
break
}
}
}
cancel() // !!This is not a graceful shutdown
println(fmt.Sprintf("%v", downstream.Trailer()))
The problem I'm having is using a context cancellation means my downstream.Trailer() response is empty. Is there a way to gracefully close this connection from the client side and receive downstream.Trailer().
Note: if I close the downstream connection from the server side, my trailers are populated. But I have no way of instructing my server side to close this particular stream. So there must be a way to gracefully close a stream client side.
Thanks.
As requested some server code :
func (b *binding) Downstream(req *pb.SimpleRequest, stream pb.Test_DownstreamServer) error {
request := req
r := make(chan *pb.SimpleResponse)
e := make(chan error)
ticker := time.NewTicker(200 * time.Millisecond)
defer func() { ticker.Stop(); close(r); close(e) }()
go func() {
defer func() { recover() }()
for {
select {
case <-ticker.C:
response, err := b.Endpoint(stream.Context(), request)
if err != nil {
e <- err
}
r <- response
}
}
}()
for {
select {
case err := <-e:
return err
case response := <-r:
if err := stream.Send(response); err != nil {
return err
}
request.Number = response.Doubled
case <-stream.Context().Done():
return nil
}
}
}
You will still need to populate the trailer with some information. I use the grpc.StreamServerInterceptor to do this.
According to the grpc go documentation
Trailer returns the trailer metadata from the server, if there is any.
It must only be called after stream.CloseAndRecv has returned, or
stream.Recv has returned a non-nil error (including io.EOF).
So if you want to read the trailer in client try something like this
ctxDownstream, cancel := context.WithCancel(ctx)
defer cancel()
for {
...
// on error or EOF
break;
}
println(fmt.Sprintf("%v", downstream.Trailer()))
Break from the infinate loop when there is a error and print the trailer. cancel will be called at the end of the function as it is deferred.
I can't find a reference that explains it clearly, but this doesn't appear to be possible.
On the wire, grpc-status is followed by the trailer metadata when the call completes normally (i.e. the server exits the call).
When the client cancels the call, neither of these are sent.
Seems that gRPC treats call cancellation as a quick abort of the rpc, not much different than the socket being dropped.
Adding a "cancel message" via request streaming works; the server can pick this up and cancel the stream from its end and trailers will still get sent:
message SimpleRequest {
oneof RequestType {
int64 number = 1;
bool cancel = 2;
}
}
....
rpc Downstream(stream SimpleRequest) returns (stream SimpleResponse);
Although this does add a bit of complication to the code.

Resources