I want to sent value from client using Context in the function contextWithValue.
func contextWithValue(key int, url string) {
req, err := http.NewRequestWithContext(contextLogger.SetValueX(context.Background()), "get", url, nil)
client := http.Client{}
resp, _ := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("Response status:", resp.Status)
scanner := bufio.NewScanner(resp.Body)
for i := 0; scanner.Scan(); i++ {
fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
panic(err)
}
}
But in the server side I cannot get the value from the context using Println function.
func Println(ctx context.Context, msg string) {
id := ctx.Value(RequestId)
if id == nil {
log.Println("Could not find value")
return
}
log.Printf("%d %s", id.(int64), msg)
}
What is the wrong here?
context is not part of HTTP as others have said. From the http doc it's stated:
For outgoing client requests, the context controls cancellation.
For incoming server requests, the context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns.
So context in term of http client-server communication is only for controlling the calcellation. For client it's to terminate request due to timeout or when context's cancel method invoked whether the request succeed or not from server side. For server it's to listen to client termination/closing connection or specifying server to timeout or stop the process if context timeout due.
You cannot pass values over context since the creation of context only happen on each side of client and server. If you want to pass values, then the only way is to create headers for them.
Related
I'm implementing a http.RoundTripper in Go, and as part of httputil.ReverseProxy implementation.
I need to buffer an incoming request, and repeat it several times, depending on the response I get from the backend. To do this, I use request.Write and http.ReadRequest. (I am actually not sure if this is a good idea, if there are any better ways, I'm interested.)
After deserializing request from []byte with http.ReadRequest and repeat it using the http.DefaultTransport’s roundtripper, I get this printed in my stderr:
2019/08/01 14:35:51 http: proxy error: unsupported protocol scheme ""
So it looks like for some reason I need to set req.URL again after deserializing to make it work.
Here's roughly how my code looks like:
func (s *myServer) RoundTrip(origReq *http.Request) (*http.Response, error) {
var b bytes.Buffer
if err := origReq.Write(&b); err != nil {
return nil, errors.Wrap(err,"failed to buffer request")
}
for retries := 0; retries < s.maxRetries; retries++{
req, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(b.Bytes()))) // probably can be simplified
if err != nil {
return nil, errors.Wrap(err,"failed to un-buffer request")
}
req.URL = origReq.URL // <-- why is this necessary?
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
if needRepeat(resp) {
continue
}
return resp, nil
}
}
ReadRequest
reads a server request. Request.Write writes a client request. See the Request.URL documentation for how the Request.URL is handled differently in client and server requests.
Given that ReadRequest and Request.Write are not inverses of each other,
a better approach is to copy the request body before the loop and create a new request on each iteration using data from the original request and the copied request body.
Summary: I'm running into a race condition during testing where my server is not reliably ready to serve requests before making client requests against it. How can I block only until the listener is ready, and still maintain composable public APIs without requiring users to BYO net.Listener?
We see the following error as the goroutine that spins up our (blocking) server in the background isn't listening before we call client.Do(req) in the TestRun test function.
--- FAIL: TestRun/Server_accepts_HTTP_requests (0.00s)
/home/matt/repos/admission-control/server_test.go:64: failed to make a request: Get https://127.0.0.1:37877: dial tcp 127.0.0.1:37877: connect: connection refused
I'm not using httptest.Server directly as I'm attempting to test the blocking & cancellation characteristics of my own server componenent.
I create an httptest.NewUnstartedServer, clone its *tls.Config into a new http.Server after starting it with StartTLS(), and then close it, before calling *AdmissionServer.Run(). This also has the benefit of giving me a *http.Client with the matching RootCAs configured.
Testing TLS is important here as the daemon this exposes lives in a TLS-only environment.
func newTestServer(ctx context.Context, t *testing.T) *httptest.Server {
testHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "OK")
})
testSrv := httptest.NewUnstartedServer(testHandler)
admissionServer, err := NewServer(nil, &noopLogger{})
if err != nil {
t.Fatalf("admission server creation failed: %s", err)
return nil
}
// We start the test server, copy its config out, and close it down so we can
// start our own server. This is because httptest.Server only generates a
// self-signed TLS config after starting it.
testSrv.StartTLS()
admissionServer.srv = &http.Server{
Addr: testSrv.Listener.Addr().String(),
Handler: testHandler,
TLSConfig: testSrv.TLS.Clone(),
}
testSrv.Close()
// We need a better synchronization primitive here that doesn't block
// but allows the underlying listener to be ready before
// serving client requests.
go func() {
if err := admissionServer.Run(ctx); err != nil {
t.Fatalf("server returned unexpectedly: %s", err)
}
}()
return testSrv
}
// Test that we can start a minimal AdmissionServer and handle a request.
func TestRun(t *testing.T) {
testSrv := newTestServer(context.TODO(), t)
t.Run("Server accepts HTTP requests", func(t *testing.T) {
client := testSrv.Client()
req, err := http.NewRequest(http.MethodGet, testSrv.URL, nil)
if err != nil {
t.Fatalf("request creation failed: %s", err)
}
resp, err := client.Do(req)
if err != nil {
t.Fatalf("failed to make a request: %s", err)
}
// Later sub-tests will test cancellation propagation, signal handling, etc.
For posterity, this is our composable Run function, that listens in a goroutine and then blocks on our cancellation & error channels in a for-select:
type AdmissionServer struct {
srv *http.Server
logger log.Logger
GracePeriod time.Duration
}
func (as *AdmissionServer) Run(ctx context.Context) error {
sigChan := make(chan os.Signal, 1)
defer close(sigChan)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
// run in goroutine
errs := make(chan error)
defer close(errs)
go func() {
as.logger.Log(
"msg", fmt.Sprintf("admission control listening on '%s'", as.srv.Addr),
)
if err := as.srv.ListenAndServeTLS("", ""); err != nil && err != http.ErrServerClosed {
errs <- err
as.logger.Log(
"err", err.Error(),
"msg", "the server exited",
)
return
}
return
}()
// Block indefinitely until we receive an interrupt, cancellation or error
// signal.
for {
select {
case sig := <-sigChan:
as.logger.Log(
"msg", fmt.Sprintf("signal received: %s", sig),
)
return as.shutdown(ctx, as.GracePeriod)
case err := <-errs:
as.logger.Log(
"msg", fmt.Sprintf("listener error: %s", err),
)
// We don't need to explictly call shutdown here, as
// *http.Server.ListenAndServe closes the listener when returning an error.
return err
case <-ctx.Done():
as.logger.Log(
"msg", fmt.Sprintf("cancellation received: %s", ctx.Err()),
)
return as.shutdown(ctx, as.GracePeriod)
}
}
}
Notes:
There is a (simple) constructor for an *AdmissionServer: I've left it out for brevity. The AdmissionServer is composable and accepts a *http.Server so that it can be plugged into existing applications easily.
The wrapped http.Server type that we create a listener from doesn't itself expose any way to tell if its listening; at best we can try to listen again and catch the error (e.g. port already bound to another listener), which does not seem robust as the net package doesn't expose a useful typed error for this.
You can just attempt to connect to the server before starting the test suite, as part of the initialization process.
For example, I usually have a function like this in my tests:
// waitForServer attempts to establish a TCP connection to localhost:<port>
// in a given amount of time. It returns upon a successful connection;
// ptherwise exits with an error.
func waitForServer(port string) {
backoff := 50 * time.Millisecond
for i := 0; i < 10; i++ {
conn, err := net.DialTimeout("tcp", ":"+port, 1*time.Second)
if err != nil {
time.Sleep(backoff)
continue
}
err = conn.Close()
if err != nil {
log.Fatal(err)
}
return
}
log.Fatalf("Server on port %s not up after 10 attempts", port)
}
Then in my TestMain() I do:
func TestMain(m *testing.M) {
go startServer()
waitForServer(serverPort)
// run the suite
os.Exit(m.Run())
}
I am looking for go code to fetch url and in most of the cases this the code for fetching url on go :
func main() {
for _, url := range os.Args[1:] {
resp, err := http.Get(url)
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: %v\n", err)
os.Exit(1)
}
b, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
fmt.Fprintf(os.Stderr, "fetch: reading %s: %v\n", url, err)
os.Exit(1)
}
fmt.Printf("%s", b)
}
}
my question why needed here resp.Body.Close() and whay this exactly doing ?
If you dig into the http docs https://golang.org/pkg/net/http/
The Get method used to make the resp is a Response
func (c *Client) Get(url string) (resp *Response, err error)
In the Response source:
// Body represents the response body.
//
// The response body is streamed on demand as the Body field
// is read. If the network connection fails or the server
// terminates the response, Body.Read calls return an error.
//
// The http Client and Transport guarantee that Body is always
// non-nil, even on responses without a body or responses with
// a zero-length body. It is the caller's responsibility to
// close Body. The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.
//
// The Body is automatically dechunked if the server replied
// with a "chunked" Transfer-Encoding.
Body io.ReadCloser
So the Close() tidies up the resources that are used to get the Body
If this isn't done the Response (resp) won't be able to do "keep-alive" and I guess there is a chance that some of the resources in the Response won't be able to be recycled
when you use ioutil.ReadAll what you are doing is calling the Read function of response.Body. What this does is set a "seeker" going which reads the contents of the response.Body and returns a buffer containing that. When you Close the response.Body you are resetting this "seeker" position at zero and returns the tcp connection to the connection pool. This ensures that any future reads you may do on this response will start from zero, as opposed to say EOF and ensures that there is no un-returned tcp connection waiting to be read.
I am playing wit golang and orientdb to test them. i have written a tiny web app which uppon a request fetches a single document from local orientdb instance and returns it. when i bench this app with apache bench, when concurrency is above 1 it get following error:
2015/04/08 19:24:07 http: panic serving [::1]:57346: Get http://localhost:2480/d
ocument/t1/9:1441: EOF
when i bench Orientdb itself, it runs perfectley OK with any cuncurrency factor.
also when i change the url to fetch from this document to anything (other program whritten in golang, some internet site etc) the app runs OK.
here is the code:
func main() {
fmt.Println("starting ....")
var aa interface{}
router := gin.New()
router.GET("/", func(c *gin.Context) {
ans := getdoc("http://localhost:2480/document/t1/9:1441")
json.Unmarshal(ans, &aa)
c.JSON(http.StatusOK, aa)
})
router.Run(":3000")
}
func getdoc(addr string) []byte {
client := new(http.Client)
req, err := http.NewRequest("GET", addr, nil)
req.SetBasicAuth("admin","admin")
resp, err := client.Do(req)
if err != nil {
fmt.Println("oops", resp, err)
panic(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
return body
}
thanks in advance
The keepalive connections are getting closed on you for some reason. You might be overwhelming the server, or going past the max number of connections the database can handle.
Also, the current http.Transport connection pool doesn't work well with synthetic benchmarks that make connections as fast as possible, and can quickly exhaust available file descriptors or ports (issue/6785).
To test this, I would set Request.Close = true to prevent the Transport from using the keepalive pool. If that works, one way to handle this with keepalive, is to specifically check for an io.EOF and retry that request, possibly with some backoff delay.
I am trying to debug a very unusual error I am receiving for a simple REST library I wrote.
I am using the standard net/http package to make Get, Post, Put, Delete requests but my tests occasionally fail when I make multiple requests successively. My test looks like this:
func TestGetObject(t *testing.T) {
firebaseRoot := New(firebase_url)
body, err := firebaseRoot.Get("1")
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
func TestPushObject(t *testing.T) {
firebaseRoot := New(firebase_url)
msg := Message{"testing", "1..2..3"}
body, err := firebaseRoot.Push("/", msg)
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
And I am making the request like this:
// Send HTTP Request, return data
func (f *firebaseRoot) SendRequest(method string, path string, body io.Reader) ([]byte, error) {
url := f.BuildURL(path)
// create a request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// send JSON to firebase
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("Bad HTTP Response: %v", resp.Status)
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return b, nil
}
Sometimes it works, but most of the time I get 1 or 2 failures:
--- FAIL: TestGetObject (0.00 seconds)
firebase_test.go:53: Error: Get https://go-firebase-test.firebaseio.com/1.json: EOF
firebase_test.go:55: ""
--- FAIL: TestPushObject (0.00 seconds)
firebase_test.go:63: Error: Post https://go-firebase-test.firebaseio.com/.json: EOF
firebase_test.go:65: ""
FAIL
exit status 1
FAIL github.com/chourobin/go.firebase 3.422s
The failures happen when I make more than 1 request. If I comment out everything except for the PUT request, the tests consistently pass. Once I include a second test, such as GET, one or the other fails (sometimes both pass).
I experienced this reliably. You need to set Req.Close to true (the defer on resp.Body.Close() syntax used in the examples is not enough). Like this:
client := &http.Client{}
req, err := http.NewRequest(method, url, httpBody)
// NOTE this !!
req.Close = true
req.Header.Set("Content-Type", "application/json")
req.SetBasicAuth("user", "pass")
resp, err := client.Do(req)
if err != nil {
// whatever
}
defer resp.Body.Close()
response, err = ioutil.ReadAll(resp.Body)
if err != nil {
// Whatever
}
I agree with the assertion that you shouldn't be hitting outside servers in your unit tests, why not just use the built-in http.Server and serve up the content that you want to test. (There is actually the httptest package to help with this)
I recently ran into this same problem while trying to crawl sitemaps, and this is what I have found so far:
Go by default will send requests with the header Connection: Keep-Alive and persist connections for re-use. The problem that I ran into is that the server is responding with Connection: Keep-Alive in the response header and then immediately closing the connection.
As a little background as to how go implements connections in this case (you can look at the full code in net/http/transport.go). There are two goroutines, one responsible for writing and one responsible for reading (readLoop and writeLoop) In most circumstances readLoop will detect a close on the socket, and close down the connection. The problem here occurs when you initiate another request before the readLoop actually detects the close, and the EOF that it reads get interpreted as an error for that new request rather than a close that occurred prior to the request.
Given that this is the case the reason why sleeping in between requests works is that it gives readLoop time to detect the close on the connection before your new request and shut it down, so that your new request will initiate a new connection. (And the reason why it would intermittently fail is because there is some amount code running between your requests and depending of scheduling of goroutines, sometimes the EOF will be properly handled before your next request, sometimes not). And the req.Close = true, solution works because it prevents the connection from being re-used.
There is a ticket related to this situation: https://code.google.com/p/go/issues/detail?id=4677 (and a dupe ticket that I created that allowed me to reliably reproduce this: https://code.google.com/p/go/issues/detail?id=8122)
I'm going to guess there is no problem with your code. The most likely cause of your problem is because the server is closing the connection. Rate limiting is one possible reason for this.
Your test shouldn't be relying on an external service that's very brittle and not hermetic. Instead you should think about spinning up a test server locally.
My experience with this error was when I entered absolutely empty input for my JSON API!
I should send {} as empty JSON, but I sent so this error happened
I encountered this issue while sending an invalid body to a GET request
I could reproduce by making a request similar to below:
var requestBody interface{}
requestData, _ := json.Marshal(requestBody)
payload = strings.NewReader(string(requestData))
req, err := http.NewRequest("GET", url, payload)
...
I was developing an image download app when this problem occurs.
Tried request.Close=true but not work.
60% requests resulted in a EOF error.
I thought maybe it is an image server problem, not my code.
But php code works fine.
Then I use
var client = &http.Client{
Transport: &http.Transport{},
}
client.Do(request)
to make request, instead of
http.DefaultClient.Do(request)
problem gone.
Not sure why,I guess something with RoundTripper