My code get file from remote url and download file in browser:
func Index(w http.ResponseWriter, r *http.Request) {
url := "http://upload.wikimedia.org/wikipedia/en/b/bc/Wiki.png"
...
resp, err := client.Get(url)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(len(body))
//download the file in browser
}
func main() {
http.HandleFunc("/", Index)
err := http.ListenAndServe(":8000", nil)
if err != nil {
fmt.Println(err)
}
}
code: http://play.golang.org/p/x-EyR2zFjv
Get file is ok, but how to downloaded it in browser?
To make the browser open the download dialog, add a Content-Disposition and Content-Type headers to the response:
w.Header().Set("Content-Disposition", "attachment; filename=WHATEVER_YOU_WANT")
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
Do this BEFORE sending the content to the client. You might also want to copy the Content-Length header of the response to the client, to show proper progress.
To stream the response body to the client without fully loading it into memory (for big files this is important) - simply copy the body reader to the response writer:
io.Copy(w, resp.Body)
io.Copy is a nice little function that take a reader interface and writer interface, reads data from one and writes it to the other. Very useful for this kind of stuff!
I've modified your code to do this: http://play.golang.org/p/v9IAu2Xu3_
In case you already have the file on disk, just use http.ServeFile(). It automatically handles Content-Length so that the browser can display a download progress.
w.Header().Set("Content-Disposition", "attachment; filename="+strconv.Quote(filename))
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeFile(w, r, filePath)
What you need to use is the Content-Disposition header with attachment value. MDN web docs says the following about the header:
[...], the Content-Disposition response header is a header indicating if the content is expected to be displayed inline in the browser, that is, as a Web page or as part of a Web page, or as an attachment, that is downloaded and saved locally.
If you want to specify a filename for the clients you can use the filename directive. In the following format:
Content-Disposition: attachment; filename="filename.jpg"
This is an optional parameter and it has some restrictions.
[...] The filename is always optional and must not be used blindly by the application: path information should be stripped, and conversion to the server file system rules should be done. [...]
To format filename properly in the header you should use mime.FormatMediaType. Example:
cd := mime.FormatMediaType("attachment", map[string]string{"filename": d.Name()})
w.Header().Set("Content-Disposition", cd)
In this case for content type you can use application/octet-stream because the browser does not have to know the MIME type of the response.
[...] Generic binary data (or binary data whose true type is unknown) is application/octet-stream. [...]
w.Header().Set("Content-Type", "application/octet-stream")
For outputing the content of a. file I would recommend to use http.ServeContent or http.ServeFile because they handle RFC 7233 - Range Requests out of the box. Example:
f, err := fs.Open(name)
// [...]
cd := mime.FormatMediaType("attachment", map[string]string{"filename": d.Name()})
w.Header().Set("Content-Disposition", cd)
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeContent(w, r, d.Name(), d.ModTime(), f)
Related
As per the changelog provided in the aws-sdk-go-v2 module we can see that they have
Disable[d] automatic decompression of getting Amazon S3 objects with the Content-Encoding: gzip metadata header.
They go on to say that you should use the aws/smithy-go's "SetHeaderValue" or "AddHeaderValue":
If you'd like the client to sent the Accept-Encoding: gzip request header, you can add this header to the API operation method call with the SetHeaderValue. middleware helper.
However, using either of those does not seem to cause the downloaded file to decompress the gzip'd file when downloading from S3. The example below shows my code which currently downloads the compressed file despite using the SetHeaderValue method suggested by AWS.
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/smithy-go/middleware"
"github.com/aws/smithy-go/transport/http"
)
func main() {
ctx := context.Background()
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
panic(err)
}
// Here I attempt to set the header at the client level
client := s3.NewFromConfig(cfg, s3.WithAPIOptions(http.SetHeaderValue("Accept-Encoding", "gzip")))
downloader := manager.NewDownloader(client, func(d *manager.Downloader) {
d.Concurrency = 1
})
fdst, err := os.Create("decompressed.txt")
if err != nil {
panic(err)
}
bucket := "bucket"
key := "test6.gz"
n, err := downloader.Download(ctx, fdst,
&s3.GetObjectInput{Bucket: &bucket, Key: &key},
// Here I attempt to set the header on a per-call basis
manager.WithDownloaderClientOptions(
func(o *s3.Options) {
o.APIOptions = append(o.APIOptions, []func(*middleware.Stack) error{
http.SetHeaderValue("Accept-Encoding", "gzip"),
}...)
},
),
)
if err != nil {
panic(err)
}
fmt.Println(n)
}
So my question is, how do I get this to actually decompress the gzip file when it downloads it? Ideally I want to control the header on a per-call basis, changing the header for the client is less useful.
downloader.Download takes a io.WriterAt. So basically everything from this Question (Buffer implementing io.WriterAt in go) works for getting the download into memory where you can decompress it.
I'm writing some middleware and I need to be able to log the response body content even when the destination is using TLS encryption.
I have a handler chain within which I store the response body in an intermediate buffer, so that I can read it more than once. This is based on the excellent example provided by icza (Golang read request body).
In my handler func, I'm doing this....
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
// Print the response body to stdout
fmt.Printf("Dest HTTP response body: %s\n", body)
bRdr := bytes.NewReader(body)
n, err := io.Copy(w, bRdr) // Copy the entire response body into our outgoing response
What I'm finding is that I get readable output when connection to a destination not using TLS, but when connected to a destination using TLS, it seems the response body is still encrypted, though the Copy into the final response to the originator results in the originator receiving valid response body content.
Is this the expected behaviour for reads of the response body with an encrypted path?
Can I decrypt this data to be able make it readable? I've read the http, tls and crypto package documentation, but have not found any clues.
I'm not sure if I understand the problem but here is me calling an https google link and printing the output.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"golang.org/x/net/http2"
)
func main() {
client := &http.Client{Transport: transport2()}
res, err := client.Get("https://www.google.com")
if err != nil {
log.Fatal(err)
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
res.Body.Close()
fmt.Printf("Code: %d\n", res.StatusCode)
fmt.Printf("Body: %s\n", body)
}
func transport2() *http2.Transport {
return &http2.Transport{
DisableCompression: true,
AllowHTTP: false,
}
}
Thanks all for your comments. Travis seems to have identified the issue I'm having. It appears the response body I'm reading is gzip encoded (the response contains "Content-Encoding: gzip"). In order to verify that this was the case, I had to explicitly remove the "Accept-Encoding: gzip" header that was in the originating request before forwarding and also configure the Transport to set "DisableCompression: true". Once I made both of those changes, I then see responses with no "Content-Encoding" header and the body I read is human readable.
I'm looking for an efficient way to read HTTP headers from a textfile to be later sent with an HTTP request. Consider the following code (which currently contains basic net/http request functionality):
func MakeRequest(target string, method string) {
client := &http.Client{}
req, _ := http.NewRequest(method, target, nil)
//Headers manually..
req.Header.Add("If-None-Match", `some value`)
response, _ := client.Do(req)
body, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(body))
}
I started by using ioutil.ReadFile like this:
func main() {
data, _ := ioutil.ReadFile("/opt/tests/req.txt")
fmt.Print(string(data))
}
But taking this text, splitting it by some indicator (lets say ":") and then placing the information in req.Header.Add("var1", "var2") per-header seems like an over-kill.
Question: Any better way to send HTTP requests with headers from a text file in go?
net/http has a method ReadRequest which can create a new Request object from a bufio.Reader. Assuming that your file contains a real HTTP request (instead of only the part of the request which consists of lines with key: value) all you need to do is create a new bufio.Reader from the file, i.e. like this (error handling omitted):
rdr,_ := os.Open("req.txt")
req,_ := http.ReadRequest(bufio.NewReader(rdr))
fmt.Printf("%+v\n", req)
If you only want some headers defined, another option is to define the headers in a Json file and apply the following code (file reading not included):
var jsonMap map[string]string
err = json.Unmarshal(jsonBytesFromFile, &jsonMap)
if err != nil {
log.Fatal("unable to parse json: ", err)
}
for k, v := range jsonMap {
log.Printf("setting Header : %s : %s", k, v)
responseWriter.Header().Add(k, v) // you may prefer Set()
}
The json looks like this:
{
"Content-type": "text/plain",
"Cache-Control": "only-if-cached"
}
I have a REST endpoint provided by a Java Spring Boot app that I am trying to hit with a simple Go (go1.8 darwin/amd64) program. I have tried to hit it in 2 separate ways:
const (
LOCAL_BASE = "http://localhost:11400"
ADD_FUNDS_ENDPOINT = "/sub-accounts/%d/add-funds"
)
type InputArgumentsJson struct {
Amount float64 `json:"amount"`
BufferAmount float64 `json:"bufferAmount"`
FromSubAccountID int `json:"fromSubAccountId"`
}
...
// set up json body
inputArguments := &InputArgumentsJson{Amount: 1100, BufferAmount: 0,
FromSubAccountID: spendFundId}
bytes := new(bytes.Buffer)
json.NewEncoder(bytes).Encode(inputArguments)
// set up the url to call
endpoint := fmt.Sprintf(ADD_FUNDS_ENDPOINT, billSetAsideId)
url := LOCAL_BASE + endpoint
client := &http.Client{}
req, err := http.NewRequest("POST", url, bytes)
if err != nil {
fmt.Println("ERROR WHILE BUILDING REQUEST")
fmt.Println(err)
}
req.Header.Add("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
fmt.Println("ERROR WHILE MAKING REQUEST")
fmt.Println(err)
}
fmt.Println(resp)
and:
resp, err := http.Post(url, "application/json; charset=utf-8", bytes)
if err != nil {
fmt.Println("ERROR WHILE MAKING REQUEST")
fmt.Println(err)
}
fmt.Println(resp)
I added a print out to see if the URL and JSON body were right and they appeared to be:
fmt.Printf("\nMaking request to %s with:\n %s\n", url, bytes)
printed out
Making request to http://localhost:11400/sub-accounts/167/add-funds with:
{"amount":1100,"bufferAmount":0,"fromSubAccountId":166}
but I am getting a 403 Forbidden when running it.
I can hit this endpoint without fail using both Postman and the following Curl command:
curl -H "Content-Type: application/json" -X POST -d '{"amount":1100,"bufferAmount":0,"fromSubAccountId":166}' http://localhost:11400/sub-accounts/167/add-funds
Does anyone have any thoughts on what I may be missing here? To add further to the intrigue I can successfully hit a different GET endpoint from the Go program.
Thanks in advance!
Edited to show that I was catching the errors and the surrounding logic for creating the URL and JSON body
You did not show what bytes is but that's most probably the problem: both NewRequest and Post take an io.Reader and read the body from that reader. This reading typically can happen only once (think of a stream). When you fmt.Printf("%s") the io.Reader this printing will consume the content (most probably by invoking a String() method) and the actual request is done with an empty body.
Always show whole code. And do not try to read twice from an io.Reader.
I assume there are 2 Go apps and runs on a different host, which results a CORS problem. Consider adding the Access-Control-Allow-Origin at your service with your client's host.
w.Header().Set("Access-Control-Allow-Origin", "*")
You can change * with specified hostname which your client application runs.
OMG....I just found the problem. It turned out that it was failing (as I was expecting), but it was erroneously returning it as a 403 rather than a 400. Ugh...sometimes when you stare at something for a long time the obvious things completely escape notice.
you should set User-Agent in request header
req, _ := http.NewRequest(http.MethodGet, urlString, nil)
req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36")
I have made an Url type that should contain the response body.
type Url struct {
Address string
Refresh string
Watch string
Found bool
Body bytes.Buffer // bytes.Buffer needs no initialization
}
An Url object is created with the right Address, and then I do
resp, err := http.Get(url.Address)
Now I would like to do something like the following, but I cannot get out of it:
io.Copy(url.Body, b) // Copy that to the Url buffer
As for now, the Url.Body field can be modified to another type if needed.
Afterwards, I want to get the string from that Buffer/Writer/whatever, but I assume this will be easy as soon as I will manage the former copy.
Regards,
Le Barde.
I guess you want to use ioutil.ReadAll which returns []byte:
resp, err := http.Get(url.Address)
if err != nil {
// handle error
}
defer resp.Body.Close()
url.Body, err = ioutil.ReadAll(resp.Body)