I want to sent value from client using Context in the function contextWithValue.
func contextWithValue(key int, url string) {
req, err := http.NewRequestWithContext(contextLogger.SetValueX(context.Background()), "get", url, nil)
client := http.Client{}
resp, _ := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("Response status:", resp.Status)
scanner := bufio.NewScanner(resp.Body)
for i := 0; scanner.Scan(); i++ {
fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
panic(err)
}
}
But in the server side I cannot get the value from the context using Println function.
func Println(ctx context.Context, msg string) {
id := ctx.Value(RequestId)
if id == nil {
log.Println("Could not find value")
return
}
log.Printf("%d %s", id.(int64), msg)
}
What is the wrong here?
context is not part of HTTP as others have said. From the http doc it's stated:
For outgoing client requests, the context controls cancellation.
For incoming server requests, the context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns.
So context in term of http client-server communication is only for controlling the calcellation. For client it's to terminate request due to timeout or when context's cancel method invoked whether the request succeed or not from server side. For server it's to listen to client termination/closing connection or specifying server to timeout or stop the process if context timeout due.
You cannot pass values over context since the creation of context only happen on each side of client and server. If you want to pass values, then the only way is to create headers for them.
I'm implementing a http.RoundTripper in Go, and as part of httputil.ReverseProxy implementation.
I need to buffer an incoming request, and repeat it several times, depending on the response I get from the backend. To do this, I use request.Write and http.ReadRequest. (I am actually not sure if this is a good idea, if there are any better ways, I'm interested.)
After deserializing request from []byte with http.ReadRequest and repeat it using the http.DefaultTransport’s roundtripper, I get this printed in my stderr:
2019/08/01 14:35:51 http: proxy error: unsupported protocol scheme ""
So it looks like for some reason I need to set req.URL again after deserializing to make it work.
Here's roughly how my code looks like:
func (s *myServer) RoundTrip(origReq *http.Request) (*http.Response, error) {
var b bytes.Buffer
if err := origReq.Write(&b); err != nil {
return nil, errors.Wrap(err,"failed to buffer request")
}
for retries := 0; retries < s.maxRetries; retries++{
req, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(b.Bytes()))) // probably can be simplified
if err != nil {
return nil, errors.Wrap(err,"failed to un-buffer request")
}
req.URL = origReq.URL // <-- why is this necessary?
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
if needRepeat(resp) {
continue
}
return resp, nil
}
}
ReadRequest
reads a server request. Request.Write writes a client request. See the Request.URL documentation for how the Request.URL is handled differently in client and server requests.
Given that ReadRequest and Request.Write are not inverses of each other,
a better approach is to copy the request body before the loop and create a new request on each iteration using data from the original request and the copied request body.
I have seen several examples where you read from r.Body and then do a defer r.Body.Close() right after. What will happen if we do not close it?
Lets say I have a http.Handler and inside it I decode the contents of r.Body like this:
func createFeedback(w http.ResponseWriter, r *http.Request) {
// ... Some code ...
f := feedback.New()
if err := json.NewDecoder(r.Body).Decode(f); err != nil {
return err
}
defer r.Body.Close()
// ... Some more code ...
}
Why do we have to close r.Body?
This is done to prevent resource leak of connections. There goes a phrase in the official go documentation:
https://pkg.go.dev/net/http#Client
If the Body is not both read to EOF and closed, the Client's
underlying RoundTripper (typically Transport) may not be able to
re-use a persistent TCP connection to the server for a subsequent
"keep-alive" request.
Does golang's net/http package support requests with chunked transfer-encoding? Thus far I have been able to use the Hijacker interface (https://golang.org/src/net/http/server.go?s=6173:6875#L156) to at least not close the connection and receive the full chunked request, but not yet parsing the chunks and suspect I may be going down the wrong path with this.
From https://golang.org/src/net/http/httputil/httputil.go?s=688:732#L10, I see there is a chunked reader, but appears to be for internal use.
Essentially, I'm trying to accept an HTTP PUT with 'chunked' transfer-encoding and send it off to a backend server 'on-the-fly' (i.e. without buffering the full request in golang). I have no control over the upstream request. Is there a recommended way to handle such a request, or is Hijacker the way to do it?
The net/http client and server transparently read and write chunked bodies.
To accept a chunked request and send it to another HTTP server, pass the server request body as the client request body. Here's now to forward the body to another server as a PUT:
func handler(w http.ResponseWriter, r *http.Request) {
creq, err := http.NewRequest("PUT", url, r.Body)
if err != nil {
// handle error
}
if ct := r.Header.Get("Content-Type"); ct != "" {
creq.Header.Set("Content-Type", ct)
}
cresp, err := http.DefaultClient.Do(creq)
if err != nil {
// handle error
}
... do something with cresp.
}
If you want to copy to a file, then io.Copy the request body to the file.
func handler(w http.ResponseWriter, r *http.Request) {
f, err := os.Create("fname")
if err != nil {
// handle error
}
_, err := io.Copy(f, r.Body)
if err != nil {
// handle error
}
...
}
These snippets copy the body 'on the fly'.
I am trying to debug a very unusual error I am receiving for a simple REST library I wrote.
I am using the standard net/http package to make Get, Post, Put, Delete requests but my tests occasionally fail when I make multiple requests successively. My test looks like this:
func TestGetObject(t *testing.T) {
firebaseRoot := New(firebase_url)
body, err := firebaseRoot.Get("1")
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
func TestPushObject(t *testing.T) {
firebaseRoot := New(firebase_url)
msg := Message{"testing", "1..2..3"}
body, err := firebaseRoot.Push("/", msg)
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
And I am making the request like this:
// Send HTTP Request, return data
func (f *firebaseRoot) SendRequest(method string, path string, body io.Reader) ([]byte, error) {
url := f.BuildURL(path)
// create a request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// send JSON to firebase
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("Bad HTTP Response: %v", resp.Status)
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return b, nil
}
Sometimes it works, but most of the time I get 1 or 2 failures:
--- FAIL: TestGetObject (0.00 seconds)
firebase_test.go:53: Error: Get https://go-firebase-test.firebaseio.com/1.json: EOF
firebase_test.go:55: ""
--- FAIL: TestPushObject (0.00 seconds)
firebase_test.go:63: Error: Post https://go-firebase-test.firebaseio.com/.json: EOF
firebase_test.go:65: ""
FAIL
exit status 1
FAIL github.com/chourobin/go.firebase 3.422s
The failures happen when I make more than 1 request. If I comment out everything except for the PUT request, the tests consistently pass. Once I include a second test, such as GET, one or the other fails (sometimes both pass).
I experienced this reliably. You need to set Req.Close to true (the defer on resp.Body.Close() syntax used in the examples is not enough). Like this:
client := &http.Client{}
req, err := http.NewRequest(method, url, httpBody)
// NOTE this !!
req.Close = true
req.Header.Set("Content-Type", "application/json")
req.SetBasicAuth("user", "pass")
resp, err := client.Do(req)
if err != nil {
// whatever
}
defer resp.Body.Close()
response, err = ioutil.ReadAll(resp.Body)
if err != nil {
// Whatever
}
I agree with the assertion that you shouldn't be hitting outside servers in your unit tests, why not just use the built-in http.Server and serve up the content that you want to test. (There is actually the httptest package to help with this)
I recently ran into this same problem while trying to crawl sitemaps, and this is what I have found so far:
Go by default will send requests with the header Connection: Keep-Alive and persist connections for re-use. The problem that I ran into is that the server is responding with Connection: Keep-Alive in the response header and then immediately closing the connection.
As a little background as to how go implements connections in this case (you can look at the full code in net/http/transport.go). There are two goroutines, one responsible for writing and one responsible for reading (readLoop and writeLoop) In most circumstances readLoop will detect a close on the socket, and close down the connection. The problem here occurs when you initiate another request before the readLoop actually detects the close, and the EOF that it reads get interpreted as an error for that new request rather than a close that occurred prior to the request.
Given that this is the case the reason why sleeping in between requests works is that it gives readLoop time to detect the close on the connection before your new request and shut it down, so that your new request will initiate a new connection. (And the reason why it would intermittently fail is because there is some amount code running between your requests and depending of scheduling of goroutines, sometimes the EOF will be properly handled before your next request, sometimes not). And the req.Close = true, solution works because it prevents the connection from being re-used.
There is a ticket related to this situation: https://code.google.com/p/go/issues/detail?id=4677 (and a dupe ticket that I created that allowed me to reliably reproduce this: https://code.google.com/p/go/issues/detail?id=8122)
I'm going to guess there is no problem with your code. The most likely cause of your problem is because the server is closing the connection. Rate limiting is one possible reason for this.
Your test shouldn't be relying on an external service that's very brittle and not hermetic. Instead you should think about spinning up a test server locally.
My experience with this error was when I entered absolutely empty input for my JSON API!
I should send {} as empty JSON, but I sent so this error happened
I encountered this issue while sending an invalid body to a GET request
I could reproduce by making a request similar to below:
var requestBody interface{}
requestData, _ := json.Marshal(requestBody)
payload = strings.NewReader(string(requestData))
req, err := http.NewRequest("GET", url, payload)
...
I was developing an image download app when this problem occurs.
Tried request.Close=true but not work.
60% requests resulted in a EOF error.
I thought maybe it is an image server problem, not my code.
But php code works fine.
Then I use
var client = &http.Client{
Transport: &http.Transport{},
}
client.Do(request)
to make request, instead of
http.DefaultClient.Do(request)
problem gone.
Not sure why,I guess something with RoundTripper