read tcp read: connection reset by peer - go

I've been using the Golang DynamoDB SDK for a while now, and recently I started seeing this error type come back:
RequestError: send request failed
caused by: Post "https://dynamodb.[REGION].amazonaws.com/": read tcp [My IP]->[AWS IP]: read: connection reset by peer
This only seems to occur when writing large amounts of data to DynamoDB, although the error is not limited to any particular type of request. I've seen it in both UpdateItem and BatchWriteItem requests. Furthermore, as the failure isn't consistent, I can't localize it to a particular line of code. It seems that the error is related to some sort of network issue between my service and AWS but, as it doesn't come back as a throttling exception, I'm not sure how to debug it. Finally, as the response comes back from a write request, I don't think retry logic is really the solution here either.
Here's my batch-write code:
func (conn *Connection) BatchWrite(tableName string, requests []*dynamodb.WriteRequest) error {
// Get the length of the requests; if there aren't any then return because there's nothing to do
length := len(requests)
log.Printf("Attempting to write %d items to DynamoDB", length)
if length == 0 {
return nil
}
// Get the number of requests to make
numRequests := length / 25
if length%25 != 0 {
numRequests++
}
// Create the variables necessary to manage the concurrency
var wg sync.WaitGroup
errs := make(chan error, numRequests)
// Attempt to batch-write the requests to DynamoDB; because DynamoDB limits the number of concurrent
// items in a batch request to 25, we'll chunk the requests into 25-report segments
sections := make([][]*dynamodb.WriteRequest, numRequests)
for i := 0; i < numRequests; i++ {
// Get the end index which is 25 greater than the current index or the end of the array
// if we're getting close
end := (i + 1) * 25
if end > length {
end = length
}
// Add to the wait group so that we can ensure all the concurrent processes finish
// before we close down the process
wg.Add(1)
// Write the chunk to DynamoDB concurrently
go func(wg *sync.WaitGroup, index int, start int, end int) {
defer wg.Done()
// Call the DynamoDB operation; record any errors that occur
if section, err := conn.batchWriteInner(tableName, requests[start:end]); err != nil {
errs <- err
} else {
sections[index] = section
}
}(&wg, i, i*25, end)
}
// Wait for all the goroutines to finish
wg.Wait()
// Attempt to read an error from the channel; if we get one then return it
// Otherwise, continue. We have to use the select here because this is
// the only way to attempt to read from a channel without it blocking
select {
case err, ok := <-errs:
if ok {
return err
}
default:
break
}
// Now, we've probably gotten retries back so take these and combine them into
// a single list of requests
retries := sections[0]
if len(sections) > 1 {
for _, section := range sections[1:] {
retries = append(retries, section...)
}
}
// Rewrite the requests and return the result
return conn.BatchWrite(tableName, retries)
}
func (conn *Connection) batchWriteInner(tableName string, requests []*dynamodb.WriteRequest) ([]*dynamodb.WriteRequest, error) {
// Create the request
request := dynamodb.BatchWriteItemInput{
ReturnConsumedCapacity: aws.String(dynamodb.ReturnConsumedCapacityNone),
ReturnItemCollectionMetrics: aws.String(dynamodb.ReturnItemCollectionMetricsNone),
RequestItems: map[string][]*dynamodb.WriteRequest{
tableName: requests,
},
}
// Attempt to batch-write the items with an exponential backoff
var result *dynamodb.BatchWriteItemOutput
err := backoff.Retry(func() error {
// Attempt the batch-write; if it fails then back-off and wait. Otherwise break out
// of the loop and return
var err error
if result, err = conn.inner.BatchWriteItem(&request); err != nil {
// If we have an error then what we do here will depend on the error code
// If the error code is for exceeded throughput, exceeded request limit or
// an internal server error then we'll try again. Otherwise, we'll break out
// because the error isn't recoverable
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case dynamodb.ErrCodeProvisionedThroughputExceededException:
case dynamodb.ErrCodeRequestLimitExceeded:
case dynamodb.ErrCodeInternalServerError:
return err
}
}
// We received an error that won't be fixed by backing off; return this as a permanent
// error so we can tell the backoff library that we want to break out of the exponential backoff
return backoff.Permanent(err)
}
return nil
}, backoff.NewExponentialBackOff())
// If the batch-write failed then return an error
if err != nil {
return nil, err
}
// Roll the unprocessed items into a single list and return them
var list []*dynamodb.WriteRequest
for _, item := range result.UnprocessedItems {
list = append(list, item...)
}
return list, nil
}
Has anyone else dealt with this issue before? What's the correct approach here?

Related

Set type of variable dynamically based on passed interface{} type

I am trying to effectively use bindings auto-generated by go-ethereum, which I stored in the uniswap_core package.
It created many different Event structs and each also has it's own Iterator struct.
Here is an example of code the tool abigen generated for an Iterator struct:
// IUniswapV3FactoryPoolCreatedIterator is returned from FilterPoolCreated and is used to iterate over the raw logs and unpacked data for PoolCreated events raised by the IUniswapV3Factory contract.
type IUniswapV3FactoryPoolCreatedIterator struct {
Event *IUniswapV3FactoryPoolCreated // Event containing the contract specifics and raw log
contract *bind.BoundContract // Generic contract to use for unpacking event data
event string // Event name to use for unpacking event data
logs chan types.Log // Log channel receiving the found contract events
sub ethereum.Subscription // Subscription for errors, completion and termination
done bool // Whether the subscription completed delivering logs
fail error // Occurred error to stop iteration
}
// Next advances the iterator to the subsequent event, returning whether there
// are any more events found. In case of a retrieval or parsing error, false is
// returned and Error() can be queried for the exact failure.
func (it *IUniswapV3FactoryPoolCreatedIterator) Next() bool {
// If the iterator failed, stop iterating
if it.fail != nil {
return false
}
// If the iterator completed, deliver directly whatever's available
if it.done {
select {
case log := <-it.logs:
it.Event = new(IUniswapV3FactoryPoolCreated)
if err := it.contract.UnpackLog(it.Event, it.event, log); err != nil {
it.fail = err
return false
}
it.Event.Raw = log
return true
default:
return false
}
}
// Iterator still in progress, wait for either a data or an error event
select {
case log := <-it.logs:
it.Event = new(IUniswapV3FactoryPoolCreated)
if err := it.contract.UnpackLog(it.Event, it.event, log); err != nil {
it.fail = err
return false
}
it.Event.Raw = log
return true
case err := <-it.sub.Err():
it.done = true
it.fail = err
return it.Next()
}
}
// Error returns any retrieval or parsing error occurred during filtering.
func (it *IUniswapV3FactoryPoolCreatedIterator) Error() error {
return it.fail
}
// Close terminates the iteration process, releasing any pending underlying
// resources.
func (it *IUniswapV3FactoryPoolCreatedIterator) Close() error {
it.sub.Unsubscribe()
return nil
}
I want to create a general function that, given any EventIterator and an interface{} channel, will iterate though the events and send them back via the channel.
A solution I've come up with is simply creating a type switch statement to get the appropriate Iterator type. But then I have to repeat the exact same iteration code within each case block.
How do I avoid this repetition?
func (f *contract_factory) readEvent(it interface{}, c_event chan interface{}) {
//TODO figure how to avoid duplicating this shit
switch iter := it.(type){
case *uniswap_core.UniswapV3FactoryPoolCreatedIterator:
for {
if ! iter.Next() {
close(c_event)
if err := iter.Error(); err != nil {
f.logger.Error("failure while iterating events", zap.Error(err))
}
break
}
c_event <- iter.Event
}
iter.Close()
case *uniswap_core.UniswapV3FactoryOwnerChangedIterator:
for {
if ! iter.Next() {
close(c_event)
if err := iter.Error(); err != nil {
f.logger.Error("failure while iterating events", zap.Error(err))
}
break
}
c_event <- iter.Event
}
iter.Close()
default:
err = errors.New("unknown event iterator type")
f.logger.Error("unknown event iterator type", zap.String("iterator type", reflect.TypeOf(iter).String()))
close(c_event)
}
}

How to pass byte slice between go routines using channels

I have a function that reads data from a source and send them to destination. Source and destination could be anything, lets say for this example source is database (any MySQL, PostgreSQL...) and destination is distributed Q (any... ActiveMQ, Kafka). Messages are stored in bytes.
This is main function. idea is it will spin a new go routine and will wait for messages to be returned for future processing.
type Message []byte
func (p *ProcessorService) Continue(dictId int) {
level.Info(p.logger).Log("process", "message", "dictId", dictId)
retrieved := make(chan Message)
go func() {
err := p.src.Read(retrieved, strconv.Itoa(p.dictId))
if err != nil {
level.Error(p.logger).Log("process", "read", "message", "err", err)
}
}()
for r := range retrieved {
go func(message Message) {
level.Info(p.logger).Log("message", message)
if len(message) > 0 {
if err := p.dst.sendToQ(message); err != nil {
level.Error(p.logger).Log("failed", "during", "persist", "err", err)
}
} else {
level.Error(p.logger).Log("failed")
}
}(r)
}
}
and this is read function itself
func (s *Storage) Read(out chan<- Message, opt ...string) error {
// I just skip some basic database read operations here
// but idea is simple, read data from the table / file row by row and
//
for _, value := range dataFromDB {
message, err := value.row
if err == nil {
out <- message
} else {
errorf("Unable to get data %v", err)
out <- make([]byte, 0)
}
}
})
close(out)
if err != nil {
return err
}
return nil
}
As you can see communication done via out chan<- Message channel.
My concern in Continue function, specifically here
for r := range retrieved {
go func(message Message) {
// basically here message and r are pointing to the same underlying array
}
}
When data received var r is a type of slice byte. Then it passed to go func(message Message) everything passed by value in go, in this case var r will be passed as copy to anonymous func, however it will still have a pointer to underlying slice data. I am curious if it could be a problem during p.dst.sendToQ(message); execution and at the same time read function will send something to out channel causing slice data structure to be overridden with a new information. Should I copy byte slice r into the new byte slice before passing to anonymous function, so underlying arrays will be different? I tested it, but couldn't really cause this behavior. Not sure if I am paranoid or have to worry about it.
The message in p.dst.sendToQ(message) is the same slice as value.row when you get data from the db. So, as long as each value.row has a different underlying array, you should be good. So, I suggest you check the source and make sure it does not use a common byte array and keeps rewriting to it.

Graceful shutdown of gRPC downstream

Using the following proto buffer code :
syntax = "proto3";
package pb;
message SimpleRequest {
int64 number = 1;
}
message SimpleResponse {
int64 doubled = 1;
}
// All the calls in this serivce preform the action of doubling a number.
// The streams will continuously send the next double, eg. 1, 2, 4, 8, 16.
service Test {
// This RPC streams from the server only.
rpc Downstream(SimpleRequest) returns (stream SimpleResponse);
}
I'm able to successfully open a stream, and continuously get the next doubled number from the server.
My go code for running this looks like :
ctxDownstream, cancel := context.WithCancel(ctx)
downstream, err := testClient.Downstream(ctxDownstream, &pb.SimpleRequest{Number: 1})
for {
responseDownstream, err := downstream.Recv()
if err != io.EOF {
println(fmt.Sprintf("downstream response: %d, error: %v", responseDownstream.Doubled, err))
if responseDownstream.Doubled >= 32 {
break
}
}
}
cancel() // !!This is not a graceful shutdown
println(fmt.Sprintf("%v", downstream.Trailer()))
The problem I'm having is using a context cancellation means my downstream.Trailer() response is empty. Is there a way to gracefully close this connection from the client side and receive downstream.Trailer().
Note: if I close the downstream connection from the server side, my trailers are populated. But I have no way of instructing my server side to close this particular stream. So there must be a way to gracefully close a stream client side.
Thanks.
As requested some server code :
func (b *binding) Downstream(req *pb.SimpleRequest, stream pb.Test_DownstreamServer) error {
request := req
r := make(chan *pb.SimpleResponse)
e := make(chan error)
ticker := time.NewTicker(200 * time.Millisecond)
defer func() { ticker.Stop(); close(r); close(e) }()
go func() {
defer func() { recover() }()
for {
select {
case <-ticker.C:
response, err := b.Endpoint(stream.Context(), request)
if err != nil {
e <- err
}
r <- response
}
}
}()
for {
select {
case err := <-e:
return err
case response := <-r:
if err := stream.Send(response); err != nil {
return err
}
request.Number = response.Doubled
case <-stream.Context().Done():
return nil
}
}
}
You will still need to populate the trailer with some information. I use the grpc.StreamServerInterceptor to do this.
According to the grpc go documentation
Trailer returns the trailer metadata from the server, if there is any.
It must only be called after stream.CloseAndRecv has returned, or
stream.Recv has returned a non-nil error (including io.EOF).
So if you want to read the trailer in client try something like this
ctxDownstream, cancel := context.WithCancel(ctx)
defer cancel()
for {
...
// on error or EOF
break;
}
println(fmt.Sprintf("%v", downstream.Trailer()))
Break from the infinate loop when there is a error and print the trailer. cancel will be called at the end of the function as it is deferred.
I can't find a reference that explains it clearly, but this doesn't appear to be possible.
On the wire, grpc-status is followed by the trailer metadata when the call completes normally (i.e. the server exits the call).
When the client cancels the call, neither of these are sent.
Seems that gRPC treats call cancellation as a quick abort of the rpc, not much different than the socket being dropped.
Adding a "cancel message" via request streaming works; the server can pick this up and cancel the stream from its end and trailers will still get sent:
message SimpleRequest {
oneof RequestType {
int64 number = 1;
bool cancel = 2;
}
}
....
rpc Downstream(stream SimpleRequest) returns (stream SimpleResponse);
Although this does add a bit of complication to the code.

Pulling 0-sized golang chan

My use-case is the following: I need to send POST requests to 0...N subscribers, which are represented by a targetUrl. I want to limitate the max number of goroutine to let's say, 100. My code (simplified) is the following:
package main
import (
"fmt"
"log"
"net/http"
"errors"
)
const MAX_CONCURRENT_NOTIFICATIONS = 100
type Subscription struct {
TargetUrl string
}
func notifySubscribers(subs []Subscription) {
log.Println("notifySubscribers")
var buffer = make(chan Subscription, len(subs))
defer close(buffer)
for i := 0; i < MAX_CONCURRENT_NOTIFICATIONS; i++ {
go notifySubscriber(buffer)
}
for i := range subs {
buffer <- subs[i]
}
}
func notifySubscriber(buffer chan Subscription) {
log.Println("notifySubscriber")
for {
select {
case sub := <-buffer:
log.Println("sending notification to " + sub.TargetUrl)
resp, err := failPost()
if err != nil {
log.Println(fmt.Sprintf("failed to notify %s. error: %s", sub.TargetUrl, err.Error()))
} else {
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Println(fmt.Sprintf("%s responded with %d", sub.TargetUrl, resp.StatusCode))
}
}
}
log.Println(fmt.Sprintf("buffer size: %d", len(buffer)))
}
}
func failPost() (*http.Response, error) {
return &http.Response{
StatusCode: http.StatusBadRequest,
}, errors.New("some bad error")
}
func main() {
log.Println("main")
var subs []Subscription
subs = append(subs, Subscription{TargetUrl: "http://foo.bar"})
subs = append(subs, Subscription{TargetUrl: "http://fizz.buzz"})
notifySubscribers(subs)
select {}
}
The output is the following:
2018/01/24 10:52:48 failed to notify . error: some bad error
2018/01/24 10:52:48 buffer size: 1
2018/01/24 10:52:48 sending notification to
2018/01/24 10:52:48 failed to notify . error: some bad error
2018/01/24 10:52:48 buffer size: 0
2018/01/24 10:52:48 sending notification to
2018/01/24 10:52:48 failed to notify . error: some bad error
... and so on till I SIGINT the program
So basically it means that I've successfuly send the notifications to the right people, but I still continue to send to empty targetUrl because I read from an empty chan.
What is wrong ?
[EDIT] Workaround, but I don't like it
for {
select {
case sub, more := <-buffer:
if !more {
return
}
}
}
It's because you are closing the buffer but your notifySubscriber is still listening on the buffer. A closed channel always returns the default type value(in this case an empty Subscription with empty TargetURL). Hence, you are getting an empty string.
Scenarios:
If you want to keep the goroutines running, then don't close the buffer.
Stop the goroutines once the work is done and then close the buffer.
From the spec:
For a channel c, the built-in function close(c) records that no more
values will be sent on the channel. It is an error if c is a
receive-only channel. Sending to or closing a closed channel causes a
run-time panic. Closing the nil channel also causes a run-time panic.
After calling close, and after any previously sent values have been
received, receive operations will return the zero value for the
channel's type without blocking. The multi-valued receive operation
returns a received value along with an indication of whether the
channel is closed.
The last sentence means that sub, more := <-buffer, more will be false if buffer is closed.
However, in your case, the code can use some improvement.
First, it makes no sense to use a select statement where there are only one case. It would just act the same without the select.
Second, in cases that the recieving channel is guaranteed to return, range over channel can be used. So your code can be changed to:
func notifySubscriber(buffer chan Subscription) {
log.Println("notifySubscriber")
for sub:= range buffer {
//Code here...
}
}

(Go) Comparing everchanging variable in ws loop

Working on a loop that receives messages and processes them accordingly, basically a websocket echo-er with keep-alives and authentication, and I've been stuck in the keep-alive part for a little while now.
The concept is simple, when the server starts I create a goroutine with a ticker, and I initialize a uint64 pointer, each time that ticker ticks (every 2 seconds), I increment the pointer with atomic.AddUint64(clockTicks, 1), then for each websocket connection goroutine, I check the variable every tick with a compare and atomic.LoadUint64(clockTicks), and I send a ping/pong message.
Edit: Seems like something is blocking the for loop until a message is received, result:
i := atomic.LoadUint64(clockTicks)
if i != cur {
cur = i
if act != true {
fmt.Println("Quit Nao U filthy bot.")
return
} else {
fmt.Println("Keep Going.")
act = false
}
}
In this snippet, i := atomic.LoadUint64(clockTicks) & all the if block only runs when i message is sent (Prints "Keep Going." on msg), which is not what I want, I want the snippet to run every for iteration, and "Keep Going." & "Quit nao ..." to trigger everytime clockTicks is incremented
Here's the important code parts, I'm using Go and Gorilla's Websockets library:
func Clock() {
clockTicks = new(uint64)
*clockTicks = 0
clock := time.NewTicker(authIntervals).C
for {
<-clock
atomic.AddUint64(clockTicks, 1)
}
}
var wsu = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: OriginHandler,
}
func serveWS(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, "Method not allowed", 405)
return
}
ws, err := wsu.Upgrade(w, r, nil)
if err != nil {
fmt.Println(err)
return
}
defer ws.Close()
cur := atomic.LoadUint64(clockTicks)
var act, val = true, false
for {
i := atomic.LoadUint64(clockTicks)
if i != cur { /* Only triggers when I receive a msg */
cur = i
if act != true {
fmt.Println("Quit Nao U filthy bot.")
return
} else {
fmt.Println("Keep Going.")
act = false
}
}
mtype, p, err := ws.ReadMessage()
if err != nil {
return
}
...
}
Edit 2: Someone in IRC suggested that maybe ws.ReadMessage is blocking, but I'm not really sure (He says ioutil.ReadAll used in ws.ReadMessage implementation is blocking it, and he's pretty sure about it)
The websocket read methods call the network connection Read method to get data from the network. Because the network connection Read method blocks, the webscocket methods also block.
To send pings, use a ticker in the write loop as in the Gorilla chat example or run a separate goroutine for pings as in Gorilla command example.

Resources