I'm writing some middleware and I need to be able to log the response body content even when the destination is using TLS encryption.
I have a handler chain within which I store the response body in an intermediate buffer, so that I can read it more than once. This is based on the excellent example provided by icza (Golang read request body).
In my handler func, I'm doing this....
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
// Print the response body to stdout
fmt.Printf("Dest HTTP response body: %s\n", body)
bRdr := bytes.NewReader(body)
n, err := io.Copy(w, bRdr) // Copy the entire response body into our outgoing response
What I'm finding is that I get readable output when connection to a destination not using TLS, but when connected to a destination using TLS, it seems the response body is still encrypted, though the Copy into the final response to the originator results in the originator receiving valid response body content.
Is this the expected behaviour for reads of the response body with an encrypted path?
Can I decrypt this data to be able make it readable? I've read the http, tls and crypto package documentation, but have not found any clues.
I'm not sure if I understand the problem but here is me calling an https google link and printing the output.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"golang.org/x/net/http2"
)
func main() {
client := &http.Client{Transport: transport2()}
res, err := client.Get("https://www.google.com")
if err != nil {
log.Fatal(err)
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
res.Body.Close()
fmt.Printf("Code: %d\n", res.StatusCode)
fmt.Printf("Body: %s\n", body)
}
func transport2() *http2.Transport {
return &http2.Transport{
DisableCompression: true,
AllowHTTP: false,
}
}
Thanks all for your comments. Travis seems to have identified the issue I'm having. It appears the response body I'm reading is gzip encoded (the response contains "Content-Encoding: gzip"). In order to verify that this was the case, I had to explicitly remove the "Accept-Encoding: gzip" header that was in the originating request before forwarding and also configure the Transport to set "DisableCompression: true". Once I made both of those changes, I then see responses with no "Content-Encoding" header and the body I read is human readable.
Related
As per the changelog provided in the aws-sdk-go-v2 module we can see that they have
Disable[d] automatic decompression of getting Amazon S3 objects with the Content-Encoding: gzip metadata header.
They go on to say that you should use the aws/smithy-go's "SetHeaderValue" or "AddHeaderValue":
If you'd like the client to sent the Accept-Encoding: gzip request header, you can add this header to the API operation method call with the SetHeaderValue. middleware helper.
However, using either of those does not seem to cause the downloaded file to decompress the gzip'd file when downloading from S3. The example below shows my code which currently downloads the compressed file despite using the SetHeaderValue method suggested by AWS.
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/smithy-go/middleware"
"github.com/aws/smithy-go/transport/http"
)
func main() {
ctx := context.Background()
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
panic(err)
}
// Here I attempt to set the header at the client level
client := s3.NewFromConfig(cfg, s3.WithAPIOptions(http.SetHeaderValue("Accept-Encoding", "gzip")))
downloader := manager.NewDownloader(client, func(d *manager.Downloader) {
d.Concurrency = 1
})
fdst, err := os.Create("decompressed.txt")
if err != nil {
panic(err)
}
bucket := "bucket"
key := "test6.gz"
n, err := downloader.Download(ctx, fdst,
&s3.GetObjectInput{Bucket: &bucket, Key: &key},
// Here I attempt to set the header on a per-call basis
manager.WithDownloaderClientOptions(
func(o *s3.Options) {
o.APIOptions = append(o.APIOptions, []func(*middleware.Stack) error{
http.SetHeaderValue("Accept-Encoding", "gzip"),
}...)
},
),
)
if err != nil {
panic(err)
}
fmt.Println(n)
}
So my question is, how do I get this to actually decompress the gzip file when it downloads it? Ideally I want to control the header on a per-call basis, changing the header for the client is less useful.
downloader.Download takes a io.WriterAt. So basically everything from this Question (Buffer implementing io.WriterAt in go) works for getting the download into memory where you can decompress it.
I am writing a program by Go.
In this program, I access to a website and in this website, it will print a string. I want to get this string for next process.
For example:
I access by curl and the returned string will like that:
curl localhost:4000
abc_example
I need to get "abc_example" for next process in my program.
Now, this problem was solved.
Actually, my result will be a JSON like that:
{"name":"xyz_example"}
How can I parse this string and just get "xyz_example"
I am a newbie in Go. May you help me.
Thank you!
Here's an example of reading the response from an HTTP request.
I would recommend reading up on the documentation for the http package, and maybe a simple tutorial like this one.
package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
//make a request
response, err := http.Get("https://mdtf.org")
if err != nil {
fmt.Println("error making request: ", err)
return
}
//make sure the response body gets closed
defer response.Body.Close()
//read the bytes
responseBytes, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Println("error reading response bytes: ", err)
return
}
//turn the response bytes into a string
responseString := string(responseBytes)
//print it or something
fmt.Println(responseString)
}
I have the following code:
package main
import (
"bytes"
"fmt"
"net/http"
)
func main() {
var jsonStr = []byte(`{"title":"Buy cheese and bread for breakfast."}`)
req, err := http.NewRequest("POST", "/", bytes.NewBuffer(jsonStr))
if err != nil {
panic(err)
}
req.Header.Set("X-Custom-Header", "myvalue")
req.Header.Set("Content-Type", "application/json")
req.ParseForm()
fmt.Printf("%v:%v", "title", req.Form.Get("title"))
}
I am unable to extract the "title" param and not sure why.
As noted in the GoDoc for the http.Request.ParseForm method, the type of the body must be application/x-www-form-urlencoded, not JSON like your current example:
For other HTTP methods, or when the Content-Type is not application/x-www-form-urlencoded, the request Body is not read, and r.PostForm is initialized to a non-nil, empty value.
Here is an updated example of your code using a form body, which gives the intended result: https://play.golang.org/p/Zrw05T2Zb5Z
If you want to extract values from a JSON body, that can be done using a method such as json.Unmarshal, however a JSON body doesn't represent a form.
The 3rd argument of http.NewRequest is the http payload.
In your case, payload type is application/json. It's need to be treated as json, only then you'll be able to get certain value from the it. In this case, we just cannot use the same technique like on getting value from query string or form data.
So just unmarshal the jsonStr data into map or struct.
res := make(map[string]interface{})
err := json.Unmarshal(jsonStr, &res)
if err != nil {
panic(err)
}
fmt.Printf("%#v \n", res["title"])
To be honest I'm quite confused with your question, why you need to get the payload from http client request.
If what you want is actually how to get the payload from the web server end, you can get it by decoding the request body. Example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
payload := make(map[string]interface{})
err := json.NewDecoder(r.Body).Decode(&payload)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
title := payload["title"].(string)
w.Write([]byte(title))
})
Curl example (based on your code):
curl -d '{"title":"Buy cheese and bread for breakfast."}' \
-H "Content-Type: application/json" \
-X POST http://localhost:9000
Output:
Buy cheese and bread for breakfast.
Because your request isn't a form. It doesn't have any GET parameters, and it isn't form-encoded data.
For other HTTP methods, or when the Content-Type is not application/x-www-form-urlencoded, the request Body is not read, and r.PostForm is initialized to a non-nil, empty value.
[ https://golang.org/pkg/net/http/#Request.ParseForm ]
You're free to parse the body of the request as application/json, but that isn't the same as form data.
The go HTTP client is adding a "chunked" transfer encoding field to my client request. Unfortunately, this is not supported by the service I'm connecting to and it comes back with an error.
Is there a way to disable this?
This is my Request code:
// DoHTTPRequest Do a full HTTP Client Request, with timeout
func DoHTTPRequest(method, url string, body io.Reader, headers map[string]string, timeout time.Duration) (*http.Response, error) {
// Create the request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// Add headers
for k, v := range headers {
req.Header.Set(k, v)
}
client := &http.Client{
Timeout: timeout,
}
return client.Do(req)
}
Basically I would like this header dropped. This client is talking to S3 which is rather sensitive as to what headers it is sent.
I get this error:
A header you provided implies functionality that is not implemented
TransferEncoding is a field directly on the Request struct. If you set it explicitly it will not be overridden.
req.TransferEncoding = []string{"identity"}
https://golang.org/pkg/net/http/#Request
Setting Transfer Encoding header like this makes it not use chunked:
req.TransferEncoding = []string{"identity"}
However the http client sources give the reason for it choosing chunked in my case. Specifically, I was using "PUT" as the method and had no content-length specified. So all I needed was to set the req.ContentLength.
However, you can see my DoHTTPRequest wrapper function doesn't know how big to set it. And I had assumed that setting the header will make it work originally. Well, it doesn't work by setting the header. And you can see why in the sources that determine whether to make it use chunked encoding.
// shouldSendChunkedRequestBody reports whether we should try to send a
// chunked request body to the server. In particular, the case we really
// want to prevent is sending a GET or other typically-bodyless request to a
// server with a chunked body when the body has zero bytes, since GETs with
// bodies (while acceptable according to specs), even zero-byte chunked
// bodies, are approximately never seen in the wild and confuse most
// servers. See Issue 18257, as one example.
//
// The only reason we'd send such a request is if the user set the Body to a
// non-nil value (say, ioutil.NopCloser(bytes.NewReader(nil))) and didn't
// set ContentLength, or NewRequest set it to -1 (unknown), so then we assume
// there's bytes to send.
//
// This code tries to read a byte from the Request.Body in such cases to see
// whether the body actually has content (super rare) or is actually just
// a non-nil content-less ReadCloser (the more common case). In that more
// common case, we act as if their Body were nil instead, and don't send
// a body.
func (t *transferWriter) shouldSendChunkedRequestBody() bool {
// Note that t.ContentLength is the corrected content length
// from rr.outgoingLength, so 0 actually means zero, not unknown.
if t.ContentLength >= 0 || t.Body == nil { // redundant checks; caller did them
return false
}
if requestMethodUsuallyLacksBody(t.Method) {
// Only probe the Request.Body for GET/HEAD/DELETE/etc
// requests, because it's only those types of requests
// that confuse servers.
t.probeRequestBody() // adjusts t.Body, t.ContentLength
return t.Body != nil
}
// For all other request types (PUT, POST, PATCH, or anything
// made-up we've never heard of), assume it's normal and the server
// can deal with a chunked request body. Maybe we'll adjust this
// later.
return true
}
So my solution is simply:
// DoHTTPRequest Do a full HTTP Client Request, with timeout
func DoHTTPRequest(method, url string, body io.Reader, headers map[string]string, timeout time.Duration) (*http.Response, error) {
// Create the request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// Add headers
for k, v := range headers {
req.Header.Set(k, v)
// Set the Content Length correctly if specified.
if strings.EqualFold(k, "Content-Length") {
length, err := strconv.Atoi(v)
if err != nil {
return nil, fmt.Errorf("Bad Content-Length header")
}
req.ContentLength = int64(length)
}
}
client := &http.Client{
Timeout: timeout,
Transport: &loghttp.Transport{},
}
return client.Do(req)
}
Which satisfies S3 as far as having a correct content length. I didn't need to set the TransferEncoding to identity.
My code get file from remote url and download file in browser:
func Index(w http.ResponseWriter, r *http.Request) {
url := "http://upload.wikimedia.org/wikipedia/en/b/bc/Wiki.png"
...
resp, err := client.Get(url)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(len(body))
//download the file in browser
}
func main() {
http.HandleFunc("/", Index)
err := http.ListenAndServe(":8000", nil)
if err != nil {
fmt.Println(err)
}
}
code: http://play.golang.org/p/x-EyR2zFjv
Get file is ok, but how to downloaded it in browser?
To make the browser open the download dialog, add a Content-Disposition and Content-Type headers to the response:
w.Header().Set("Content-Disposition", "attachment; filename=WHATEVER_YOU_WANT")
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
Do this BEFORE sending the content to the client. You might also want to copy the Content-Length header of the response to the client, to show proper progress.
To stream the response body to the client without fully loading it into memory (for big files this is important) - simply copy the body reader to the response writer:
io.Copy(w, resp.Body)
io.Copy is a nice little function that take a reader interface and writer interface, reads data from one and writes it to the other. Very useful for this kind of stuff!
I've modified your code to do this: http://play.golang.org/p/v9IAu2Xu3_
In case you already have the file on disk, just use http.ServeFile(). It automatically handles Content-Length so that the browser can display a download progress.
w.Header().Set("Content-Disposition", "attachment; filename="+strconv.Quote(filename))
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeFile(w, r, filePath)
What you need to use is the Content-Disposition header with attachment value. MDN web docs says the following about the header:
[...], the Content-Disposition response header is a header indicating if the content is expected to be displayed inline in the browser, that is, as a Web page or as part of a Web page, or as an attachment, that is downloaded and saved locally.
If you want to specify a filename for the clients you can use the filename directive. In the following format:
Content-Disposition: attachment; filename="filename.jpg"
This is an optional parameter and it has some restrictions.
[...] The filename is always optional and must not be used blindly by the application: path information should be stripped, and conversion to the server file system rules should be done. [...]
To format filename properly in the header you should use mime.FormatMediaType. Example:
cd := mime.FormatMediaType("attachment", map[string]string{"filename": d.Name()})
w.Header().Set("Content-Disposition", cd)
In this case for content type you can use application/octet-stream because the browser does not have to know the MIME type of the response.
[...] Generic binary data (or binary data whose true type is unknown) is application/octet-stream. [...]
w.Header().Set("Content-Type", "application/octet-stream")
For outputing the content of a. file I would recommend to use http.ServeContent or http.ServeFile because they handle RFC 7233 - Range Requests out of the box. Example:
f, err := fs.Open(name)
// [...]
cd := mime.FormatMediaType("attachment", map[string]string{"filename": d.Name()})
w.Header().Set("Content-Disposition", cd)
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeContent(w, r, d.Name(), d.ModTime(), f)