I have a http request which I need to inspect the body of. But when I do, the request fails. I'm assuming this had to do with the Reader needing to be reset, but googling along the lines of go ioutil reset ReadCloser hasn't turned anything up that looks promising.
c is a *middleware.Context,
c.Req.Request is a http.Request, and
c.Req.Request.Body is an io.ReadCloser
contents, _ := ioutil.ReadAll(c.Req.Request.Body)
log.Info("Request: %s", string(contents))
proxy.ServeHTTP(c.RW(), c.Req.Request)
Specifically the error I get is http: proxy error: http: ContentLength=133 with Body length 0
You can't reset it, because you've already read from it and there's nothing left in the stream.
What you can do is take the buffered bytes you already have, and replace the Body with a new io.ReadCloser
contents, _ := ioutil.ReadAll(c.Req.Request.Body)
log.Info("Request: %s", string(contents))
c.Req.Request.Body = ioutil.NopCloser(bytes.NewReader(contents))
proxy.ServeHTTP(c.RW(), c.Req.Request)
Related
I'm writing a webhook in Go that parses a JSON payload. I'm attempting to log the raw payload and then decode it immediately after but it fails when I try. If I perform the actions separately, they both work fine independently.
Can someone explain why I can't use ioutil.ReadAll and json.NewDecoder together?
func webhook(w http.ResponseWriter, r *http.Request) {
body, _ := ioutil.ReadAll(r.Body)
log.Printf("incoming message - %s", body)
var p payload
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&p)
if err != nil {
// Returns EOF
log.Printf("invalid payload - %s", err)
}
defer r.Body.Close()
}
Can someone explain why I can't use ioutil.ReadAll and json.NewDecoder
together?
The request body is an io.ReadCloser that reads bytes, more or less, directly from a network connection. The contents of the Body aren't stored in memory by default. That's why after the first time you've read the Body the next time you try to read it you'll get EOF.
So if you need to process the request Body more than once, you yourself will have to store the contents into memory, which is what you are already doing with:
body, _ := ioutil.ReadAll(r.Body)
You can then reuse body as many times as you like, and since you have the Body contents at your disposal as a []byte value, you can use json.Unmarshal instead of json.NewDecoder(...).Decode.
This is unrelated to your question, but please do not ignore the error returned from ioutil.ReadAll.
Also you can drop the defer r.Body.Close() line, because you do not have to close the request body in your server handlers. (emphasis mine)
For server requests the Request Body is always non-nil but will return
EOF immediately when no body is present. The Server will close the
request body. The ServeHTTP Handler does not need to.
r.Body is meant to be read exactly once.
When you use the ioutil.ReadAll function you do read all the data from the body. That's why the decoder which also relies on r.Body in fact gets nothing to decode.
Minor additional point about json.Decoder and json.Unmarshal: at first glance it looks like the only difference between the two is just that the former operates on a stream and the latter on a []byte, but they actually have different semantics.
json.Unmarshal will return an error if the data contains more than one json object. So, for example, it will parse {}, but it will not parse {}{}.
json.Decoder parses one complete object per call to Decode, so if you give it {}{}, it will parse those two objects and then the third call will return io.EOF and it's More method will return false.
In a normal http body, you probably only want a single object, so you'd want to use Unmarshal if you're not worried about loading all the data into memory at once. You can also use Decoder and manually check that there is only one object if you care to do so.
I have created a simple reverse proxy in my main.go as follows:
reverseproxy := httputil.NewSingleHostReverseProxy("https://someurl/someuri")
this is working fine, however, I would like to log the response that comes back from the server. I can also do this utilizing the following code inside the standard RoundTrip method recommended by golang:
response, err := http.DefaultTransport.RoundTrip(request)
dumpresp, err := httputil.DumpResponse(response, true)
if err != nil {
return nil, err
}
log.Printf("%s", dumpresp)
All the above works as expected aside from one thing, the response, when Content-Encoding: gzip, the string appears to the logs as non-utf8 gzip characters. it seems to be a golang oversight but maybe there is something i have missed in the documentation, which I have read to its completion a few times. I can't post the logs here because the characters are non utf8 so they would not display on this site anyway. So, I know what your thinking, just grab the gzip content and use a method to remove the gzip compression. That would be great if the response from DumpResponse was not partly gzip and partly standard utf8 with no way to separate the sections from each other.
So I know what your going to say, why not just take the raw response like the following and gzip decode, and "not" use DumpResponse. Well I can do that as well, but there is an issue with the following:
var reader io.Reader
startreader := httputility.NewChunkedReader(reader)
switch response.Header.Get("Content-Encoding") {
case "gzip":
startreader, err = gzip.NewReader(response.Body)
log.Println("Response body gzip: ")
buf := new(bytes.Buffer)
buf.ReadFrom(startreader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
for name, value := range response.Header {
var hvaluecomp string = ""
for i := 0; i < len(value); i++ {
hvaluecomp += value[i]
}
response.Header.Add(name,hvaluecomp)
}
log.Printf("%s", s)
default:
startreader = response.Body
buf := new(bytes.Buffer)
buf.ReadFrom(startreader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
for name, value := range response.Header {
fmt.Printf("%v: %v\n", name, value)
}
log.Printf("%s", s)
}
The issue with the above is, responses can only be read one time via the reader, after that the response cannot be read by the reverse proxy anymore and it makes the proxy response back to the browser nil =), so again I was met with failure. I cant imagine that the folks coding golang would have missed decoding gzip, just strange, it seems to be so simple.
So in the end, my question is, does DumpResponse give me the ability to decompress gzip so I can log the actual response instead of non utf8 characters? This does the log reader no good when debugging an issue for the production product. This would render the built in golang reverse proxy useless in my eyes and I would start development on my own.
The answer is to copy the stream, then you can use the second variable to re-post the stream back to the request.body object so you don't lose any data.
buf, _ := ioutil.ReadAll(response.Body)
responseuse1 := ioutil.NopCloser(bytes.NewBuffer(buf))
responsehold := ioutil.NopCloser(bytes.NewBuffer(buf))
Method to pull logging: extractLogging(responseuse1)
Push the hold back to the body so its untouched: response.Body = responsehold
return response
does DumpResponse give me the ability to decompress gzip
No it does not.
To me it seems as if the major problem is neither the reverse proxy nor DumpResponse but that you are trying to "log" binary data: Be it gzip, or other binary data like images. Just fix your logging logic: If the raw body is binary you should render some kind of representation or transformation of it. For gziped stuff gunzip it first (but this might still be binary "non utf8" data unsuitable for logging). Focus on the real problem: How to "log" binary data.
I have a client application which reads in the full body of a http response into a buffer and performs some processing on it:
body, _ = ioutil.ReadAll(containerObject.Resp.Body)
The problem is that this application runs on an embedded device, so responses that are too large fill up the device RAM, causing Ubuntu to kill the process.
To avoid this, I check the content-length header and bypass processing if the document is too large. However, some servers (I'm looking at you, Microsoft) send very large html responses without setting content-length and crash the device.
The only way I can see of getting around this is to read the response body up to a certain length. If it reaches this limit, then a new reader could be created which first streams the in-memory buffer, then continues reading from the original Resp.Body. Ideally, I would assign this new reader to the containerObject.Resp.Body so that callers would not know the difference.
I'm new to GoLang and am not sure how to go about coding this. Any suggestions or alternative solutions would be greatly appreciated.
Edit 1: The caller expects a Resp.Body object, so the solution needs to be compatible with that interface.
Edit 2: I cannot parse small chunks of the document. Either the entire document is processed or it is passed unchanged to the caller, without loading it into memory.
If you need to read part of the response body, then reconstruct it in place for other callers, you can use a combination of an io.MultiReader and ioutil.NopCloser
resp, err := http.Get("http://google.com")
if err != nil {
return err
}
defer resp.Body.Close()
part, err := ioutil.ReadAll(io.LimitReader(resp.Body, maxReadSize))
if err != nil {
return err
}
// do something with part
// recombine the buffered part of the body with the rest of the stream
resp.Body = ioutil.NopCloser(io.MultiReader(bytes.NewReader(part), resp.Body))
// do something with the full Response.Body as an io.Reader
If you can't defer resp.Body.Close() because you intend to return the response before it's read in its entirety, you will need to augment the replacement body so that the Close() method applies to the original body. Rather than using the ioutil.NopCloser as the io.ReadCloser, create your own that refers to the correct method calls.
type readCloser struct {
io.Closer
io.Reader
}
resp.Body = readCloser{
Closer: resp.Body,
Reader: io.MultiReader(bytes.NewReader(part), resp.Body),
}
Creating a byte array of a http request and then trying to read it into a http.request doesn't seem to work when the request includes a body.
req, _ := http.NewRequest(http.MethodPost, "/Bar", strings.NewReader("Foo"))
rReq, _ := httputil.DumpRequest(req, true)
req2, _ := http.ReadRequest(bufio.NewReader(bytes.NewReader(rReq)))
b, _ := ioutil.ReadAll(req2.Body)
fmt.Println(b)
b is an empty array.
Two things are wrong in your code:
You must handle the errors. This would have helpesd you see that you never construct a valid request ("/Bar" is not a valid URL).
Use httputil.DumpRequestOut for outgoing request.
Takeaways: Always handle all errors and always read the whole whole package doc.
I reading big file and sending this file by http POST.
I use bufio.
And now I want to modify one of first line of this file, how to do it ?
f := bufio.NewReaderSize(os.Stdin, 65536)
bufPart, err := f.Peek(65536))
//how to modify bufPart(f) ?
...
req, err := http.NewRequest("POST", url, f)
Two ideas how to do it:
Create your own Reader implementation that wraps an bufio.Reader and implements replacing logic (you will have to count number of read bytes).
Call io.Pipe, pass the returned PipeReader to NewRequest and start a separate goroutine that will read data from a file, modify it and write to the returned PipeWriter.
Hope this makes sense.