I'm using GO redis client redigo to write image to ~20 redis servers.
speed is an important factor here and I'm just sending set commands to the redis so I'm using Send and Flush without calling Receive.
after a few hours I'm getting "connection reset by peer" on the client.
I was wondering, does it have something to do with the fact that I don't call Receive?
maybe my RX queue just getting to its max capacity because I don't empty it with Receive?
Thank you.
An application must call Receive to clear the responses from the server and to check for errors. If the application is not pipelining commands, then it's best to call Do. Do combines Send, Flush and Receive.
If you don't care about errors, then start a goroutine to read the responses:
go func(c redis.Conn) {
for c.Err() == nil {
c.Receive()
}
}()
Related
I'm working with Gorilla Websocket and curious about how the WriteMessage and ReadMessage functions work.
Does the WriteMessage function send the bytes data to the client synchronously? Or Does the ReadMessage actively fetch the data from the server (according to the documentation, we need to create an event loop to call the ReadMessage function).
What happened if the server kept calling WriteMessage, but no one read the message (the client calls the ReadMessage function through event loop), is the data is lost, or is it kept until the next read request came? Thank You.
Does the WriteMessage function send the bytes data to the client synchronously?
WriteMessage writes the data to the underlying network connection.
The operating system network connection maintains a buffer of data to transmit to the peer. Data is removed from the buffer when the peer acknowledges that the peer received data.
Write to the operating system network connection returns after all of the application data is added to the buffer. Write can block waiting for space in the buffer.
It is almost always the case that the application write call returns before the peer receives the data. A successful call to WriteMessage does imply that the peer application read the data.
Or Does the ReadMessage actively fetch the data from the server (according to the documentation, we need to create an event loop to call the ReadMessage function).
ReadMessage calls read on the underlying network connection.
The operating system buffers some amount of data received from peer.
Read on the operating network connection blocks until data is available in the buffer.
What happened if the server kept calling WriteMessage, but no one read the message.
WriteMessage will eventually block waiting for space in the operating system transmit buffer.
Use a write deadline to protect against blocking forever on a dead or stuck peer.
is the data is lost, or is it kept until the next read request came?
The data is held in operating system transmit and receive buffers.
Application write to the websocket connection blocks when the transmit buffer is full.
The data is only lost if the peer application terminates before the peer application reads the data.
You can find source code for that function here: https://github.com/gorilla/websocket/blob/c3dd95aea9779669bb3daafbd84ee0530c8ce1c1/conn.go#L751-L774
And it looks like this is is blocking/sync method.
As per tracking They creating writer here: https://github.com/gorilla/websocket/blob/c3dd95aea9779669bb3daafbd84ee0530c8ce1c1/conn.go#L766
w, err := c.NextWriter(messageType)
Then they are writing the data:
if _, err = w.Write(data); err != nil {
return err
}
And this is blocking because they are closing connection in the last line of that function, so writing must be done at this moment.
This is behavior of io.WriteCloser interface returned into w variable.
What happened if the server kept calling WriteMessage, but no one read the message (the client calls the ReadMessage function through event loop), is the data is lost, or is it kept until the next read request came? Thank You.
You should set Write/Read timeouts.
Library is not repeating sending the data for you. You need to implement this logic in your application.
If server is up and receive your connection, (probably) it will read your message(if it is not stopped before perform your data).
If you sent message and server was dead(did not receive your message), your data is lost.
Additional reference:
The w.Write function: https://github.com/gorilla/websocket/blob/c3dd95aea9779669bb3daafbd84ee0530c8ce1c1/conn.go#L650-L675
The io.WriteCloser interface desc: https://golang.org/pkg/io/#WriteCloser
Gorila Websocket timeouts: https://pkg.go.dev/github.com/gorilla/websocket#Conn.SetReadDeadline
Timeouts documentation for Gorila: https://pkg.go.dev/github.com/gorilla/websocket#Conn.SetReadDeadline
I plan on having two services.
HTTP REST service written in Ruby
JSON RPC service written in Go
The Ruby service will open a TCP socket connection to a Go JSON RPC service. It'll do this for each incoming HTTP request it receives. It will send some data over the socket to the Go service and that service will subsequently send back the corresponding data back down the socket.
Go code
The Go service go would look something like this (simplified):
srv := new(service.App) // this would expose a Process method
rpc.Register(srv)
listener, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
for {
conn, err := listener.Accept()
if err != nil {
// handle error
}
go jsonrpc.ServeConn(conn)
}
Notice we serve the incoming connection using a goroutine, so we can handle requests concurrently.
Ruby code
Below is a simple snippet of Ruby code that demonstrates (in theory) the way I would send data to the Go service:
require "socket"
require "json"
socket = TCPSocket.new "localhost", "8080"
b = {
:method => "App.Process",
:params => [{ :Config => JSON.generate({ :foo => :bar }) }],
:id => "0"
}
socket.write(JSON.dump(b))
response = JSON.load socket.readline
My concern is: will this be a safe sequence of events?
I'm not asking if this will be 'thread safe', because i'm not worried about manipulating shared memory across the go routines. I'm more concerned around whether my Ruby HTTP service will get back the data it's expecting?
If I have two parallel requests coming into my HTTP Service (or maybe the Ruby app is hosted behind a load balancer and so different instances of the HTTP service is handling multiple requests), then I could have instance A send the message Foo to the Go service; while instance B sends the message Bar.
The business logic inside the Go service will return different responses depending on its input so I want to be sure that Ruby instance A gets back the correct response for Foo, and B gets back the correct response for Bar.
I assume a socket connection is more like a queue in that if instance A makes a request to the Go service first and then B does, but B is quicker responding for whatever reason, then the Go service will write the response for B to the socket and instance A of the Ruby app will end up reading in the wrong socket data (this is obviously just one possible scenario considering that I could get lucky and have instance B read the socket data before instance A does).
Solutions?
I'm not sure if there is simple solution to this problem. Unless I don't use a TCP socket or RPC and instead rely on standard HTTP in the Go service. But I wanted the performance and less overhead of TCP.
I'm worried the design could get more complicated by maybe having to implement an external queue as a way of synchronising the responses with the Ruby service.
It maybe because the nature of my Ruby service is fundamentally synchronous (HTTP response/request) that I have no option but to switch to HTTP for the Go service.
But wanted to double check with the community first just in case I'm missing something obvious.
Yes this is safe if you create a new connection every time.
That said there are latent issues with your approach:
TCP connections are rather expensive to establish, so you probably want to re-use connections with a connection pool
If you make too many simultaneous requests you will exhaust ports/open file descriptors which will cause your program to crash
You don't have any timeouts in place, so it's possible to end up with orphaned TCP connections which never complete (either because of something bad on the Go side, or network problems)
I think you'd be better off using HTTP (despite the overhead) since libraries are already written to cope with these problems. HTTP is also much more debuggable since you can just curl an endpoint to test it.
Personally I'd probably go with gRPC.
I have a simple multipart form which uploads to a Go app. I wanted to set a restriction on the upload size, so I did the following:
func myHandler(rw http.ResponseWriter, request *http.Request){
request.Body = http.MaxBytesReader(rw, request.Body, 1024)
err := request.ParseMultipartForm(1024)
if err != nil{
// Some response.
}
}
Whenever an upload exceeds the maximum size, I get a connection reset like the following:
and yet the code continues executing. I can't seem to provide any feedback to the user. Instead of severing the connection I'd prefer to say "You've exceeded the size limit". Is this possible?
This code works as intended. Description of http.MaxBytesReader
MaxBytesReader is similar to io.LimitReader but is intended for
limiting the size of incoming request bodies. In contrast to
io.LimitReader, MaxBytesReader's result is a ReadCloser, returns a
non-EOF error for a Read beyond the limit, and closes the underlying
reader when its Close method is called.
MaxBytesReader prevents clients from accidentally or maliciously
sending a large request and wasting server resources.
You could use io.LimitReader to read just N bytes and then do the handling of the HTTP request on your own.
The only way to force a client to stop sending data is to forcefully close the connection, which is what you're doing with http.MaxBytesReader.
You could use a io.LimitReader wrapped in a ioutil.NopCloser, and notify the client of the error state. You could then check for more data, and try and drain the connection up to another limit to keep it open. However, clients that aren't responding correctly to MaxBytesReader may not work in this case either.
The graceful way to handle something like this is using Expect: 100-continue, but that only really applies to clients other than web browsers.
I'm new to concurrent programming, and have no idea what concepts to start with, so please be gentle.
I am writing a webservice as a front-end to a TCP server. This server listens to the port I give it, and returns the response to the TCP connection for each request.
Here is why I'm writing a web-service front-end for this server:
The server can handle one request at a time, and I'm trying to make it be able to process several inputs concurrently, by launching multiple processes and giving them a different port to listen on. For example, I want to launch 30 instances and tell them to listen on ports 20000-20029.
Our team uses PHP, and PHP does not have the capacity to launch server instances and maintain them concurrently, so I'm trying to write an API they can just send HTTP requests to.
So, here is the structure I have thought of.
I will have a main() function. This function launches the processes concurrently, then starts an HTTP server on port 80 and listens.
I have an http.Handler that adds the content of a request to a channel,.
I will have gorutines, one per server instance, that are in an infinite loop.
The code for the function mentioned in item three would be something like this:
func handleRequest(queue chan string) {
for {
request := <-queue
conn, err := connectToServer()
err = sendRequestToServer(conn)
response, err := readResponseFromServer(conn)
}
}
So, my http.Handler can simply do something like queue<- request to add the request to the queue, and handleRequest, which has blocked, waiting for the channel to have something to get, will simply get the request and continue on. When done, the loop finishes, execution comes back to the request := <-queue, and the same thing continues.
My problem starts in the http.Handler. It makes perfect sense to put requests in a channel, because multiple gorutines are all listening to it. However, how can these gorutines return the result to my http.Handler?
One way is to use a channel, let's call it responseQueue, that all of these gorutines would then write to. The problem is that when a response is added to the channel, I don't know which request it belongs to. In other words, when multiple http.Handlers send requests, each executing handler will not know which response the current message in the channel belongs to.
Is there a best practice, or a pattern, to send data to a gorutine from another gorutine and receive the data back?
Create a per request response channel and include it in the value sent to the worker. The handler receives from the channel. The worker sends the result to the channel.
Is there a concept of acknowledgements in Redis Pub/Sub?
For example, when using RabbitMQ, I can have two workers running on separate machines and when I publish a message to the queue, only one of the workers will ack/nack it and process the message.
However I have discovered with Redis Pub/Sub, both workers will process the message.
Consider this simple example, I have this go routine running on two different machines/clients:
go func() {
for {
switch n := pubSubClient.Receive().(type) {
case redis.Message:
process(n.Data)
case redis.Subscription:
if n.Count == 0 {
return
}
case error:
log.Print(n)
}
}
}()
When I publish a message:
conn.Do("PUBLISH", "tasks", "task A")
Both go routines will receive it and run the process function.
Is there a way of achieving similar behaviour to RabbitMQ? E.g. first worker to ack the message will be the only one to receive it and process it.
Redis PubSub is more like a broadcast mechanism.
if you want queues, you can use BLPOP along with RPUSH to get the same interraction. Keep in mind, RabbitMQ does all sorts of other stuff that are not really there in Redis. But if you looking for simple job scheduling / request handling style, this will work just fine.
No, Redis' PubSub does not guarantee delivery nor does it limit the number of possible subscribers who'll get the message.
Redis streams (now, with Redis 5.0) support acknowledgment of tasks as they are completed by a group.
https://redis.io/topics/streams-intro