Go http server bad performance on request forwarding - performance

Question

Hi I developed a little go server that does (at the moment) nothing but forwarding the request to a local service on the machine it is running.
So nearly the same as nginx as reverse proxy.
But I observed a really bad performance that even uses up all resources of the server and runs into timeouts on further requests.
I know that this cannot be as performant as nginx, but I don't think that it should be that slow.
Here is the server I use for forwarding the request:
package main
import (
"github.com/gorilla/mux"
"net/http"
"github.com/sirupsen/logrus"
"bytes"
"io/ioutil"
)
func main() {
router := mux.NewRouter()
router.HandleFunc("/", forwarder).Methods("POST")
server := http.Server{
Handler: router,
Addr: ":8443",
}
logrus.Fatal(server.ListenAndServeTLS("cert.pem", "key.pem"))
}
var client = &http.Client{}
func forwarder(w http.ResponseWriter, r *http.Request) {
// read request
body, err := ioutil.ReadAll(r.Body)
if err != nil {
logrus.Error(err.Error())
ServerError(w, nil)
return
}
// create forwarding request
req, err := http.NewRequest("POST", "http://localhost:8000", bytes.NewReader(body))
if err != nil {
logrus.Error(err.Error())
ServerError(w, nil)
return
}
resp, err := client.Do(req)
if err != nil {
logrus.Error(err.Error())
ServerError(w, nil)
return
}
// read response
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
logrus.Error(err.Error())
ServerError(w, nil)
return
}
resp.Body.Close()
// return response
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(resp.StatusCode)
w.Write(respBody)
}
From the client side I just measure the roundtrip time. And when I fire 100 Requests per second the response time goes up quite fast.
It starts with a response time of about 50ms. After 10 Seconds the response time is at 500ms. After 10 more seconds the response time is at 8000ms and so on, until I get timeouts.
When I use the nginx instead of my server there is no problem running 100 requests per second. Using nginx it stays at 40ms per each request.
Some observation:
using nginx: lsof -i | grep nginx
has no more than 2 connections open.
using my server the number of connection increases up to 500 and then the connections with state SYN_SENT increases and then the requets run into timeouts.
Another finding: I measured the delay of this code line:
resp, err := client.Do(req)
There is where most of the time is spent, but the could also just be because the go routines are starving!?
What I also tried:
r.Close = true (or KeepAlive = false)
I modified timeouts on the server side
I modified all this stuff on the http client used by my forward server (keepalive false, request.Close = true) etc.
I don't know why I got such a bad performance.
My guess is that go runs into problems because of the huge number of go routines. Maybe most of the time is used up scheduling this go routines and so the latency goes up?
I also tried to use the included httputil.NewSingleHostReverseProxy(). Performance is a little bit better, but still the same problem.
UPDATE:
Now I tried fasthttp:
package main
import (
"github.com/sirupsen/logrus"
"github.com/valyala/fasthttp"
)
func StartNodeManager() {
fasthttp.ListenAndServeTLS(":8443", "cert.pem", "key.pem", forwarder)
}
var client = fasthttp.Client{}
func forwarder(ctx *fasthttp.RequestCtx) {
resp := fasthttp.AcquireResponse()
req := fasthttp.AcquireRequest()
req.Header.SetMethod("POST")
req.SetRequestURI("http://127.0.0.1:8000")
req.SetBody(ctx.Request.Body())
err := client.Do(req, resp)
if err != nil {
logrus.Error(err.Error())
ctx.Response.SetStatusCode(500)
return
}
ctx.Response.SetBody(resp.Body())
fasthttp.ReleaseRequest(req)
fasthttp.ReleaseResponse(resp)
}
Little bit better but after 30 seconds the first timeouts arrive and the response time goes up to 5 seconds.

Answer 1

The root cause of the problem is GO http module is not handling connections to upstream in
a manged way, time is increasing because lots of connections are getting opened and they go into time_wait state.
So with number of increasing connections, you will get decrease in performance.
You just have to set
// 1000 what I am using
http.DefaultTransport.(*http.Transport).MaxIdleConns = 1000
http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = 1000
in your forwarder and this will solve your problem.
By the way, use go std library reverse proxy, this will take away lot of headache.
But still for reverse proxy you need to set MaxIdleConns and MaxIdleConnsPerHost , in it's transport.
Follow the article given below.

Answer 2

First of all you should profile your app and find out where is the bottleneck.
Second I would be looking to way write code with less memory allocation in heap and more on stack.
Few ideas:
Do you need read request body for all request?
Do you need always read response body?
Can you pass body of client request to request to server? func NewRequest(method, url string, body io.Reader) (*Request, error)
Use sync.Pool
Consider using fasthttp as it creates less pressure to garbage collector
Check if your server uses same optimisation as Nginx. E.g. Keep-Alive, caching, etc.
Again profile and compare against Nginx.
Seems there is a lot of space for optimization.