Is there a way to do a client.Do("POST", "example.com", body) and read the response headers before the entire response body has been received/closed? This would be similar to how JavaScript XHR requests emit an event that the headers have been received and you can read them before the rest of the request arrives.
What I'm trying to accomplish is making a sort of "smart client" that uses information in the headers from my server to determine what to upload in the request body. So I need to start the request, read the response headers, then start writing to the request body. Because of the nature of my system, I can't split it across separate requests. I believe it's possible at the protocol level, but I'm not sure if go's http libraries support it.
http client Do function doesn't block until whole response body is returned. if you don't want to read full response, why not just use res.Body.Close() after you have examined headers?. i think it should work if you want roughly same behavior. According to Doc.
The response body is streamed on demand as the Body field is read. If the network
connection fails or the server terminates the response, Body.Read calls return an error.
Although DefaultTransport of default http.Client which is http.Transport doesn't give you guarantee that it won't read any byte before you specify.
You can fulfill your requirements by sending an OPTIONS request to the url before sending actual request and read the response header.
The response header will contain all the necessary headers to perform the preferred request.
req, _ := http.NewRequest("OPTIONS", "example.com", nil)
resp, _ := client.Do(req)
Related
When I read the source code of the Golang net/http package, it always mentioned the difference between client requests and server requests. I want to know when these two are generally used, (of course, client requests are well understood and are widely used in websites with front-end and back-end separation architectures)
e.g.
//
// For client requests, a nil body means the request has no
// body, such as a GET request. The HTTP Client's Transport
// is responsible for calling the Close method.
//
// For server requests, the Request Body is always non-nil
// but will return EOF immediately when no body is present.
// The Server will close the request body. The ServeHTTP
// Handler does not need to.
//
// Body must allow Read to be called concurrently with Close.
// In particular, calling Close should unblock a Read waiting
// for input.
Body io.ReadCloser
You can create a Request using http.NewRequest, then set its fields, and call Client.Do to issue the request. In this case, the Request.Body is a variable the client has to set if the request has a body.
When this request is handled on the server side, a new instance of http.Request is created, and that is a server request. For this use, the Body is never nil.
There is this excellent blog post by Jack Lindamood How to correctly use context.Context in Go 1.7 which boils down to the following money quote:
Context.Value should inform, not control. This is the primary mantra that I feel should guide if you are using context.Value correctly. The
content of context.Value is for maintainers not users. It should never
be required input for documented or expected results.
Currently, I am using Context to transport the following information:
RequestID which is generated on the client-side passed to the Go backend and it solely travels through the command-chain and is then inserted in the response again. Without the RequestID in the response, the client-side would break though.
SessionID identifies the WebSocket session, this is important when certain responses are generated in asynchronous computations (e.g. worker queues) in order to identify on which WebSocket session the response should be send.
When taking the definition very seriously I would say both violate the intention of context.Context but then again their values do not change any behavior while the whole request is made, it's only relevant when generating the response.
What's the alternative? Having the context.Context for metadata in the server API actually helps to maintain lean method signatures because this data is really irrelevant to the API but only important for the transport layer which is why I am reluctant to create something like a request struct:
type Request struct {
RequestID string
SessionID string
}
and make it part of every API method which solely exists to be passed through before sending a response.
Based on my understanding context should be limited to passing things like request or session ID. In my application, I do something like below in one of my middleware. Helps with observability
if next != nil {
if requestID != "" {
b := context.WithValue(r.Context(), "requestId", requestID)
r = r.WithContext(b)
}
next.ServeHTTP(w, r)
}
I'm listening to Eventbus as below and calling HttpClient.postAbs() using vert.x
public void start(Future<Void> fut) {
EventBus eb1 = MainAdminVx.serviceBack.getEventBus();
eb1.consumer("local-message-receiver", message -> {
HttpClient client = vertx.createHttpClient();
client.postAbs("http://external-server-address/test#xyz.com/activityIn?activityId=5", r -> {
r.bodyHandler(b -> System.out.println(b.toString() + r.statusCode() )
).exceptionHandler(t -> System.err.println(t.getMessage()));
})
.putHeader("content-length", "1000")
.putHeader("userId", "test#xyz.com")
.putHeader("Content-Type", "application/json")
.putHeader("Accept", "application/json")
.write("some text")
.exceptionHandler(System.err::println)
.end();
});
}
Is there anything I'm missing? or there is another way to do this... because I'm getting response after 2mins in postAbs and same POST request is working quickly in postman .
Thanks in advance !!!
We just have to set
setChunked(true)
and it works like a charm !!!
As I have set content-length as '1000' , so server will wait until it receives 1000 bytes even though your not sending anything and respond back after. It means the exact byte length of the HTTP body. Generally it is used for HTTP 1.1 so that the receiving party knows when the current response/request has finished, so the connection can be reused for another request.
So what is Chunked transfer :
Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which data is sent in a series of "chunks". It uses the Transfer-Encoding HTTP header in place of the Content-Length header, which the earlier version of the protocol would otherwise require. Because the Content-Length header is not used, the sender does not need to know the length of the content before it starts transmitting a response to the receiver. Senders can begin transmitting dynamically-generated content before knowing the total size of that content.
Alternatively, content-length can be omitted and a chunked encoding can be used, or if both are missing, then at the end of the response the connection must be closed.
I hope this will help you to understand Http chunk concept.
If I have a basic http handler for POST requests, how can I stop processing if the payload is larger than 100 KB?
From what I understand, in my POST Handler, behind the scenes the server is streaming the POSTED data. But if I try and access it, it will block correct?
I want to stop processing if it is over 100 KB in size.
Use http.MaxBytesReader to limit the amount of data read from the client. Execute this line of code
r.Body = http.MaxBytesReader(w, r.Body, 100000)
before calling r.ParseForm, r.FormValue or any other request method that reads the body.
Wrapping the request body with io.LimitedReader limits the amount of data read by the application, but does not necessarily limit the amount of data read by the server on behalf of the application.
Checking the request content length is unreliable because the field is not set to the actual request body size when chunked encoding is used.
I believe you can simply check http.Request.ContentLength param to know about the size of the posted request prior to decide whether to go ahead or return error if larger than expected.
I have a function that just makes a get request to check the status code. It does not read anything from the body. Should I still end the function with resp.Body.Close() ?
Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.
Yes. When you call http.Get, the function returns a response as soon as all the HTTP headers have been read. The body of the response has not been read yet. The Response.Body is a wrapper around the network connection to the server. When you read from it, it downloads the body of the response.
.Close() tells the system that you're done with the network connection. If you have not read the response body, the default http transport closes the connection. (The transport can only re-use the connection if the body has been read, because if it reused a connection with an unread body the next request made using that connection would receive the previous request's response!)
So reading the Body is often more efficient than simply Close()ing if you're making more than one request - especially with TLS connections which are relatively expensive to create.
If you don't need the body of the response, you should use Head instead of Get. Head doesn't require reading or closing the response body.