I'm implementing a http.RoundTripper in Go, and as part of httputil.ReverseProxy implementation.
I need to buffer an incoming request, and repeat it several times, depending on the response I get from the backend. To do this, I use request.Write and http.ReadRequest. (I am actually not sure if this is a good idea, if there are any better ways, I'm interested.)
After deserializing request from []byte with http.ReadRequest and repeat it using the http.DefaultTransport’s roundtripper, I get this printed in my stderr:
2019/08/01 14:35:51 http: proxy error: unsupported protocol scheme ""
So it looks like for some reason I need to set req.URL again after deserializing to make it work.
Here's roughly how my code looks like:
func (s *myServer) RoundTrip(origReq *http.Request) (*http.Response, error) {
var b bytes.Buffer
if err := origReq.Write(&b); err != nil {
return nil, errors.Wrap(err,"failed to buffer request")
}
for retries := 0; retries < s.maxRetries; retries++{
req, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(b.Bytes()))) // probably can be simplified
if err != nil {
return nil, errors.Wrap(err,"failed to un-buffer request")
}
req.URL = origReq.URL // <-- why is this necessary?
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
if needRepeat(resp) {
continue
}
return resp, nil
}
}
ReadRequest
reads a server request. Request.Write writes a client request. See the Request.URL documentation for how the Request.URL is handled differently in client and server requests.
Given that ReadRequest and Request.Write are not inverses of each other,
a better approach is to copy the request body before the loop and create a new request on each iteration using data from the original request and the copied request body.
Related
I just started to use Golang and I want to remake my already working NodeJS/TypeScript app in Go.
One endpoint of my API simply adds server-side generated authorization headers and sends a request to a remote API. Basically filling those headers for me by calling my API instead of the remote API.
This is what I am currently writing
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-Id", "---------")
client := &http.Client{}
res, err := client.Do(req)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
var forwardedBody interface{}
json.Unmarshal(body, &forwardedBody)
return ctx.Status(fiber.StatusOK).JSON(forwardedBody)
}
I'd like to know if I am on the right steps, because making a request, parsing the JSON response with ioutil then unmarshall it to send it back seems kind of overboard for the simplicity of what I am trying to achieve ?
Edit: Thank you for the help, this is what I will be going for
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-ID", "---------")
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return ctx.SendStatus(fiber.StatusBadRequest)
}
ctx.Set("Content-Type", "application/json; charset=utf-8")
return ctx.Status(res.StatusCode).SendStream(res.Body)
}
You can use httputil.ReverseProxy. Which takes a base URL and forwards requests to the base URL, concatenating the path.
ReverseProxy is an HTTP Handler that takes an incoming request and sends it to another server, proxying the response back to the client.
http.Handle("/", &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme = "https"
r.URL.Host = "go.dev"
r.Host = r.URL.Host
r.Header.Set("X-Foo", "Bar")
},
})
If you are not serving this from the root path / you can use StripPrefix.
http.HandleFunc("/foo/", http.StripPrefix("/foo/", proxy)
There is also a helper function NewSingleHostReverseProxy, which possibly removes the need to configure the proxy struct yourself. But I think it will be better to set the Host header along with your custom header.
You don't need to attempt to parse the data as JSON. This will be problematic if any of your endpoints don't return JSON, anyway, so just inject the body directly into the response:
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
// Inject the body from the inner response into the actual response so it can be returned
ctx.Response().SetBody(body)
return cx.Status(fiber.StatusOK)
I am looking for go code to fetch url and in most of the cases this the code for fetching url on go :
func main() {
for _, url := range os.Args[1:] {
resp, err := http.Get(url)
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: %v\n", err)
os.Exit(1)
}
b, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: reading %s: %v\n", url, err)
os.Exit(1)
}
fmt.Printf("%s", b)
}
}
my question why needed here resp.Body.Close() and whay this exactly doing ?
If you dig into the http docs https://golang.org/pkg/net/http/
The Get method used to make the resp is a Response
func (c *Client) Get(url string) (resp *Response, err error)
In the Response source:
// Body represents the response body.
//
// The response body is streamed on demand as the Body field
// is read. If the network connection fails or the server
// terminates the response, Body.Read calls return an error.
//
// The http Client and Transport guarantee that Body is always
// non-nil, even on responses without a body or responses with
// a zero-length body. It is the caller's responsibility to
// close Body. The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.
//
// The Body is automatically dechunked if the server replied
// with a "chunked" Transfer-Encoding.
Body io.ReadCloser
So the Close() tidies up the resources that are used to get the Body
If this isn't done the Response (resp) won't be able to do "keep-alive" and I guess there is a chance that some of the resources in the Response won't be able to be recycled
when you use ioutil.ReadAll what you are doing is calling the Read function of response.Body. What this does is set a "seeker" going which reads the contents of the response.Body and returns a buffer containing that. When you Close the response.Body you are resetting this "seeker" position at zero and returns the tcp connection to the connection pool. This ensures that any future reads you may do on this response will start from zero, as opposed to say EOF and ensures that there is no un-returned tcp connection waiting to be read.
Does golang's net/http package support requests with chunked transfer-encoding? Thus far I have been able to use the Hijacker interface (https://golang.org/src/net/http/server.go?s=6173:6875#L156) to at least not close the connection and receive the full chunked request, but not yet parsing the chunks and suspect I may be going down the wrong path with this.
From https://golang.org/src/net/http/httputil/httputil.go?s=688:732#L10, I see there is a chunked reader, but appears to be for internal use.
Essentially, I'm trying to accept an HTTP PUT with 'chunked' transfer-encoding and send it off to a backend server 'on-the-fly' (i.e. without buffering the full request in golang). I have no control over the upstream request. Is there a recommended way to handle such a request, or is Hijacker the way to do it?
The net/http client and server transparently read and write chunked bodies.
To accept a chunked request and send it to another HTTP server, pass the server request body as the client request body. Here's now to forward the body to another server as a PUT:
func handler(w http.ResponseWriter, r *http.Request) {
creq, err := http.NewRequest("PUT", url, r.Body)
if err != nil {
// handle error
}
if ct := r.Header.Get("Content-Type"); ct != "" {
creq.Header.Set("Content-Type", ct)
}
cresp, err := http.DefaultClient.Do(creq)
if err != nil {
// handle error
}
... do something with cresp.
}
If you want to copy to a file, then io.Copy the request body to the file.
func handler(w http.ResponseWriter, r *http.Request) {
f, err := os.Create("fname")
if err != nil {
// handle error
}
_, err := io.Copy(f, r.Body)
if err != nil {
// handle error
}
...
}
These snippets copy the body 'on the fly'.
Here my use case
We have one services "foobar" which has two version legacy and version_2_of_doom (both in go)
In order to make the transition from legacy to version_2_of_doom , we would like in a first time, to have the two version alongside, and have the POST request (as there's only one POST api call in this ) received on both.
The way I see how to do it. Would be
modifying the code of legacy at the beginning of the handler, in order to duplicate the request to version_2_of_doom
func(w http.ResponseWriter, req *http.Request) {
req.URL.Host = "v2ofdoom.local:8081"
req.Host = "v2ofdoom.local:8081"
client := &http.Client{}
client.Do(req)
// legacy code
but it seems to not be as straightforward as this
it fails with http: Request.RequestURI can't be set in client requests.
Is there a well-known method to do this kind of action (i.e transfering without touching) a http.Request to an other server ?
You need to copy the values you want into a new request. Since this is very similar to what a reverse proxy does, you may want to look at what "net/http/httputil" does for ReverseProxy.
Create a new request, and copy only the parts of the request you want to send to the next server. You will also need to read and buffer the request body if you intend to use it both places:
func handler(w http.ResponseWriter, req *http.Request) {
// we need to buffer the body if we want to read it here and send it
// in the request.
body, err := ioutil.ReadAll(req.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// you can reassign the body if you need to parse it as multipart
req.Body = ioutil.NopCloser(bytes.NewReader(body))
// create a new url from the raw RequestURI sent by the client
url := fmt.Sprintf("%s://%s%s", proxyScheme, proxyHost, req.RequestURI)
proxyReq, err := http.NewRequest(req.Method, url, bytes.NewReader(body))
// We may want to filter some headers, otherwise we could just use a shallow copy
// proxyReq.Header = req.Header
proxyReq.Header = make(http.Header)
for h, val := range req.Header {
proxyReq.Header[h] = val
}
resp, err := httpClient.Do(proxyReq)
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
defer resp.Body.Close()
// legacy code
}
In my experience, the easiest way to achieve this was to simply create a new request and copy all request attributes that you need into the new request object:
func(rw http.ResponseWriter, req *http.Request) {
url := req.URL
url.Host = "v2ofdoom.local:8081"
proxyReq, err := http.NewRequest(req.Method, url.String(), req.Body)
if err != nil {
// handle error
}
proxyReq.Header.Set("Host", req.Host)
proxyReq.Header.Set("X-Forwarded-For", req.RemoteAddr)
for header, values := range req.Header {
for _, value := range values {
proxyReq.Header.Add(header, value)
}
}
client := &http.Client{}
proxyRes, err := client.Do(proxyReq)
// and so on...
This approach has the benefit of not modifying the original request object (maybe your handler function or any middleware functions that are living in your stack still need the original object?).
Using original request (copy or duplicate only if original request still need):
func handler(w http.ResponseWriter, r *http.Request) {
// Step 1: rewrite URL
URL, _ := url.Parse("https://full_generic_url:123/x/y")
r.URL.Scheme = URL.Scheme
r.URL.Host = URL.Host
r.URL.Path = singleJoiningSlash(URL.Path, r.URL.Path)
r.RequestURI = ""
// Step 2: adjust Header
r.Header.Set("X-Forwarded-For", r.RemoteAddr)
// note: client should be created outside the current handler()
client := &http.Client{}
// Step 3: execute request
resp, err := client.Do(r)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Step 4: copy payload to response writer
copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
resp.Body.Close()
}
// copyHeader and singleJoiningSlash are copy from "/net/http/httputil/reverseproxy.go"
func copyHeader(dst, src http.Header) {
for k, vv := range src {
for _, v := range vv {
dst.Add(k, v)
}
}
}
func singleJoiningSlash(a, b string) string {
aslash := strings.HasSuffix(a, "/")
bslash := strings.HasPrefix(b, "/")
switch {
case aslash && bslash:
return a + b[1:]
case !aslash && !bslash:
return a + "/" + b
}
return a + b
}
I've seen the accepted anwser, but I would like to say that I dont like this. I've used this code for months with it working, but after some time you encounter requests that break (POST requests in my case). My preferred solution is the following:
r.URL.Host = "example.com"
r.RequestURI = ""
client := &http.Client{}
delete(r.Header, "Accept-Encoding")
delete(r.Headers, "Content-Length")
resp, err := client.Do(r.WithContext(context.Background())
if err != nil {
return nil, err
}
return resp, nil
I am trying to debug a very unusual error I am receiving for a simple REST library I wrote.
I am using the standard net/http package to make Get, Post, Put, Delete requests but my tests occasionally fail when I make multiple requests successively. My test looks like this:
func TestGetObject(t *testing.T) {
firebaseRoot := New(firebase_url)
body, err := firebaseRoot.Get("1")
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
func TestPushObject(t *testing.T) {
firebaseRoot := New(firebase_url)
msg := Message{"testing", "1..2..3"}
body, err := firebaseRoot.Push("/", msg)
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
And I am making the request like this:
// Send HTTP Request, return data
func (f *firebaseRoot) SendRequest(method string, path string, body io.Reader) ([]byte, error) {
url := f.BuildURL(path)
// create a request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// send JSON to firebase
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("Bad HTTP Response: %v", resp.Status)
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return b, nil
}
Sometimes it works, but most of the time I get 1 or 2 failures:
--- FAIL: TestGetObject (0.00 seconds)
firebase_test.go:53: Error: Get https://go-firebase-test.firebaseio.com/1.json: EOF
firebase_test.go:55: ""
--- FAIL: TestPushObject (0.00 seconds)
firebase_test.go:63: Error: Post https://go-firebase-test.firebaseio.com/.json: EOF
firebase_test.go:65: ""
FAIL
exit status 1
FAIL github.com/chourobin/go.firebase 3.422s
The failures happen when I make more than 1 request. If I comment out everything except for the PUT request, the tests consistently pass. Once I include a second test, such as GET, one or the other fails (sometimes both pass).
I experienced this reliably. You need to set Req.Close to true (the defer on resp.Body.Close() syntax used in the examples is not enough). Like this:
client := &http.Client{}
req, err := http.NewRequest(method, url, httpBody)
// NOTE this !!
req.Close = true
req.Header.Set("Content-Type", "application/json")
req.SetBasicAuth("user", "pass")
resp, err := client.Do(req)
if err != nil {
// whatever
}
defer resp.Body.Close()
response, err = ioutil.ReadAll(resp.Body)
if err != nil {
// Whatever
}
I agree with the assertion that you shouldn't be hitting outside servers in your unit tests, why not just use the built-in http.Server and serve up the content that you want to test. (There is actually the httptest package to help with this)
I recently ran into this same problem while trying to crawl sitemaps, and this is what I have found so far:
Go by default will send requests with the header Connection: Keep-Alive and persist connections for re-use. The problem that I ran into is that the server is responding with Connection: Keep-Alive in the response header and then immediately closing the connection.
As a little background as to how go implements connections in this case (you can look at the full code in net/http/transport.go). There are two goroutines, one responsible for writing and one responsible for reading (readLoop and writeLoop) In most circumstances readLoop will detect a close on the socket, and close down the connection. The problem here occurs when you initiate another request before the readLoop actually detects the close, and the EOF that it reads get interpreted as an error for that new request rather than a close that occurred prior to the request.
Given that this is the case the reason why sleeping in between requests works is that it gives readLoop time to detect the close on the connection before your new request and shut it down, so that your new request will initiate a new connection. (And the reason why it would intermittently fail is because there is some amount code running between your requests and depending of scheduling of goroutines, sometimes the EOF will be properly handled before your next request, sometimes not). And the req.Close = true, solution works because it prevents the connection from being re-used.
There is a ticket related to this situation: https://code.google.com/p/go/issues/detail?id=4677 (and a dupe ticket that I created that allowed me to reliably reproduce this: https://code.google.com/p/go/issues/detail?id=8122)
I'm going to guess there is no problem with your code. The most likely cause of your problem is because the server is closing the connection. Rate limiting is one possible reason for this.
Your test shouldn't be relying on an external service that's very brittle and not hermetic. Instead you should think about spinning up a test server locally.
My experience with this error was when I entered absolutely empty input for my JSON API!
I should send {} as empty JSON, but I sent so this error happened
I encountered this issue while sending an invalid body to a GET request
I could reproduce by making a request similar to below:
var requestBody interface{}
requestData, _ := json.Marshal(requestBody)
payload = strings.NewReader(string(requestData))
req, err := http.NewRequest("GET", url, payload)
...
I was developing an image download app when this problem occurs.
Tried request.Close=true but not work.
60% requests resulted in a EOF error.
I thought maybe it is an image server problem, not my code.
But php code works fine.
Then I use
var client = &http.Client{
Transport: &http.Transport{},
}
client.Do(request)
to make request, instead of
http.DefaultClient.Do(request)
problem gone.
Not sure why,I guess something with RoundTripper