I want to make a hello world web app that will properly catch template error. So I need to buffer the response but don't know how to do it. I've put together this code. Is this the way to buffer a response in golang?
func get_handler(w http.ResponseWriter, r *http.Request) {
buf := new(bytes.Buffer)
err := templates.ExecuteTemplate(buf, "hello.html", nil)
if err != nil {
fmt.Println(err.Error())
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Write([]byte(buf.String()))
}
bytes.Buffer has a Bytes method, so you don't actually need to call String and convert it to []byte:
w.Write(buf.Bytes())
Additionally, it's a good practice to write errors to stderr. Just replace your fmt with log:
if err != nil {
log.Println(err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
bytes.Buffer has a built-in method for writing out to an io.Writer:
func get_handler(w http.ResponseWriter, r *http.Request) {
buf := new(bytes.Buffer)
//fill buffer
_,err := buf.WriteTo(w)
if err != nil {
log.Println(err)
}
}
docs: http://golang.org/pkg/bytes/#Buffer.WriteTo
You can use a bytes.Buffer "as is" by writing into it, but at the cost of discarding the buffer on every request.
Using a pooled approach (that safely resets the contents between use, but retains the buffer) can have some gains:
var bufPool *bpool.BufferPool
func YourHandler(w http.ResponseWriter, r *http.Request) {
buf := bufPool.Get()
defer bufPool.Put(buf)
err := template.ExecuteTemplate(buf, "forms/create.html", user)
// or err := json.NewEncoder(buf).Encode(value)
if err != nil {
return err
}
buf.WriteTo(w)
}
This uses the bpool library.
Related
I've been playing around with the spotify api and came to an Problem. context.Context gets used and therefore the functions just "randomly" execute. The OAuth function should check if the Code is invalid but If I don't do this with an channel the last part of the code gets executed directly without even the first/second function finishing. Because of that I made an second goroutine that checks if the channel is received and then write an response. But now I get this error http: wrote more than the declared Content-Length how can I correct the Content-Lenght? Why is context even used?
My Code:
// Wrapper: github.com/zmb3/spotify/v2
func WriteResponse(w http.ResponseWriter, h chan *spotify.Client) {
client := <-h
user, err := client.CurrentUser(context.Background())
fmt.Println(user.User.DisplayName)
if err != nil {
_, err := fmt.Fprint(w, "Couldn't get user sorry :(")
if err != nil {
return
}
}
_, err = fmt.Fprintf(w, "Logged in as %s!", user.User.DisplayName)
if err != nil {
log.Println(err)
return
}
}
func OAuth(w http.ResponseWriter, r *http.Request) {
ch := make(chan *spotify.Client)
tok, err := auth.Token(r.Context(), state, r)
if err != nil {
w.WriteHeader(503)
_, err := fmt.Fprint(w, "Couldn't get token sorry :(")
if err != nil {
return
}
}
if st := r.FormValue("state"); st != state {
http.NotFound(w,r)
log.Fatalf("State mismatch: %s != %s\n", st, state)
}
go WriteResponse(w, ch)
client := spotify.New(auth.Client(r.Context(), tok))
ch <- client
}
You forgot to return..
if err != nil {
w.WriteHeader(503)
_, err := fmt.Fprint(w, "Couldn't get token sorry :(")
if err != nil {
return
}
// here
return
}
I'm trying to test a Go function which performs a call to an external service. Here's the function:
func (gs *EuGameService) retrieveGames(client model.HTTPClient) (model.EuGamesResponse, error) {
req, err := http.NewRequest(http.MethodGet, gs.getGamesEndpoint, nil)
if err != nil {
log.Fatal("Error while creating request ", err)
return nil, err
}
resp, err := client.Do(req)
if err != nil {
log.Fatal("Error while retrieving EU games", err)
return nil, err
}
var euGames model.EuGamesResponse
decoder := json.NewDecoder(resp.Body)
decoder.Decode(&euGames)
return euGames, nil
}
to properly test it, I'm trying to inject a mock client.
type HTTPClient interface {
Do(req *http.Request) (*http.Response, error)
}
type mockClient struct{}
func (mc *mockClient) Do(req *http.Request) (*http.Response, error) {
mock, _ := os.Open("../stubs/eugames.json")
defer mock.Close()
r := ioutil.NopCloser(bufio.NewReader(mock))
return &http.Response{
Status: string(http.StatusOK),
StatusCode: http.StatusOK,
Body: r,
}, nil
}
the file eugames.json contains a couple of games. But for some reason, the body is always empty! What am I missing here? I tried to use a constant with the file content and it works, games are decoded correctly. So I'm assuming there's a problem with my use of the file.
How do I set the Request.FormFile when trying to test an endpoint?
Partial code:
func (a *EP) Endpoint(w http.ResponseWriter, r *http.Request) {
...
x, err := strconv.Atoi(r.FormValue("x"))
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
f, fh, err := r.FormFile("y")
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
...
}
How do I use the httptest lib to generate a post request that has value that I can get in FormFile?
You don't need to mock the complete FormFile struct as suggested by the other answer. The mime/multipart package implements a Writer type that lets you create a FormFile. From the docs
CreateFormFile is a convenience wrapper around CreatePart. It creates
a new form-data header with the provided field name and file name.
func (w *Writer) CreateFormFile(fieldname, filename string) (io.Writer, error)
Then, you can pass this io.Writer to httptest.NewRequest, which accepts a reader as an argument.
request := httptest.NewRequest("POST", "/", myReader)
To do this, you can either write the FormFile to an io.ReaderWriter buffer or use an io.Pipe. Here is a complete example that makes use of pipes:
func TestUploadImage(t *testing.T) {
// Set up a pipe to avoid buffering
pr, pw := io.Pipe()
// This writer is going to transform
// what we pass to it to multipart form data
// and write it to our io.Pipe
writer := multipart.NewWriter(pw)
go func() {
defer writer.Close()
// We create the form data field 'fileupload'
// which returns another writer to write the actual file
part, err := writer.CreateFormFile("fileupload", "someimg.png")
if err != nil {
t.Error(err)
}
// https://yourbasic.org/golang/create-image/
img := createImage()
// Encode() takes an io.Writer.
// We pass the multipart field
// 'fileupload' that we defined
// earlier which, in turn, writes
// to our io.Pipe
err = png.Encode(part, img)
if err != nil {
t.Error(err)
}
}()
// We read from the pipe which receives data
// from the multipart writer, which, in turn,
// receives data from png.Encode().
// We have 3 chained writers!
request := httptest.NewRequest("POST", "/", pr)
request.Header.Add("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
handler := UploadFileHandler()
handler.ServeHTTP(response, request)
t.Log("It should respond with an HTTP status code of 200")
if response.Code != 200 {
t.Errorf("Expected %s, received %d", 200, response.Code)
}
t.Log("It should create a file named 'someimg.png' in uploads folder")
if _, err := os.Stat("./uploads/someimg.png"); os.IsNotExist(err) {
t.Error("Expected file ./uploads/someimg.png' to exist")
}
}
This function makes use of the image package to generate a file dynamically taking advantage of the fact that you can pass an io.Writer to png.Encode. In the same vein, you could pass your multipart Writer to generate the bytes in a CSV format (NewWriter in package "encoding/csv"), generating a file on the fly, without needing to read anything from your filesystem.
If you have a look at the implementation of the FormFile function you'll see that it reads the exposed MultipartForm field.
https://golang.org/src/net/http/request.go?s=39022:39107#L1249
// FormFile returns the first file for the provided form key.
1258 // FormFile calls ParseMultipartForm and ParseForm if necessary.
1259 func (r *Request) FormFile(key string) (multipart.File, *multipart.FileHeader, error) {
1260 if r.MultipartForm == multipartByReader {
1261 return nil, nil, errors.New("http: multipart handled by MultipartReader")
1262 }
1263 if r.MultipartForm == nil {
1264 err := r.ParseMultipartForm(defaultMaxMemory)
1265 if err != nil {
1266 return nil, nil, err
1267 }
1268 }
1269 if r.MultipartForm != nil && r.MultipartForm.File != nil {
1270 if fhs := r.MultipartForm.File[key]; len(fhs) > 0 {
1271 f, err := fhs[0].Open()
1272 return f, fhs[0], err
1273 }
1274 }
1275 return nil, nil, ErrMissingFile
1276 }
In your test you should be able to create a test instance of multipart.Form and assign it to your request object - https://golang.org/pkg/mime/multipart/#Form
type Form struct {
Value map[string][]string
File map[string][]*FileHeader
}
Of course this will require that you use a real filepath which isn't great from a testing perspective. To get around this you could define an interface to read FormFile from a request object and pass a mock implementation into your EP struct.
Here is a good post with a few examples on how to do this: https://husobee.github.io/golang/testing/unit-test/2015/06/08/golang-unit-testing.html
I combined these and other answers into an Echo example without pipes or goroutines:
func Test_submitFile(t *testing.T) {
path := "testfile.txt"
body := new(bytes.Buffer)
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("object", path)
assert.NoError(t, err)
sample, err := os.Open(path)
assert.NoError(t, err)
_, err = io.Copy(part, sample)
assert.NoError(t, err)
assert.NoError(t, writer.Close())
e := echo.New()
req := httptest.NewRequest(http.MethodPost, "/", body)
req.Header.Set(echo.HeaderContentType, writer.FormDataContentType())
rec := httptest.NewRecorder()
c := e.NewContext(req, rec)
c.SetPath("/submit")
if assert.NoError(t, submitFile(c)) {
assert.Equal(t, 200, rec.Code)
assert.Contains(t, rec.Body.String(), path)
fi, err := os.Stat(expectedPath)
if os.IsNotExist(err) {
t.Fatal("Upload file does not exist", expectedPath)
}
assert.Equal(t, wantSize, fi.Size())
}
}
By combining the previous answers, this worked for me:
filePath := "file.jpg"
fieldName := "file"
body := new(bytes.Buffer)
mw := multipart.NewWriter(body)
file, err := os.Open(filePath)
if err != nil {
t.Fatal(err)
}
w, err := mw.CreateFormFile(fieldName, filePath)
if err != nil {
t.Fatal(err)
}
if _, err := io.Copy(w, file); err != nil {
t.Fatal(err)
}
// close the writer before making the request
mw.Close()
req := httptest.NewRequest(http.MethodPost, "/upload", body)
req.Header.Add("Content-Type", mw.FormDataContentType())
res := httptest.NewRecorder()
// router is of type http.Handler
router.ServeHTTP(res, req)
I'm currently developing a download server in Go. I need to limit the download speed of users to 100KB/s.
This was my code:
func serveFile(w http.ResponseWriter, r *http.Request) {
fileID := r.URL.Query().Get("fileID")
if len(fileID) != 0 {
w.Header().Set("Content-Disposition", "attachment; filename=filename.txt")
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
w.Header().Set("Content-Length", r.Header.Get("Content-Length"))
file, err := os.Open(fmt.Sprintf("../../bin/files/test.txt"))
defer file.Close()
if err != nil {
http.NotFound(w, r)
return
}
io.Copy(w, file)
} else {
io.WriteString(w, "Invalid request.")
}
}
Then I found a package on github and my code became the following:
func serveFile(w http.ResponseWriter, r *http.Request) {
fileID := r.URL.Query().Get("fileID")
if len(fileID) != 0 {
w.Header().Set("Content-Disposition", "attachment; filename=Wiki.png")
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
w.Header().Set("Content-Length", r.Header.Get("Content-Length"))
file, err := os.Open(fmt.Sprintf("../../bin/files/test.txt"))
defer file.Close()
if err != nil {
http.NotFound(w, r)
return
}
bucket := ratelimit.NewBucketWithRate(100*1024, 100*1024)
reader := bufio.NewReader(file)
io.Copy(w, ratelimit.Reader(reader, bucket))
} else {
io.WriteString(w, "Invalid request.")
}
}
But I'm getting this error:
Corrupted Content Error
The page you are trying to view cannot be shown because an error in
the data transmission was detected.
Here's my code on the Go playground: http://play.golang.org/p/ulgXQl4eQO
Rather than mucking around with getting the correct the content type and length headers yourself it'd probably be much better to use http.ServeContent which will do that for you (as well as support "If-Modified-Since", range requests, etc. If you can supply an "ETag" header it can also handle "If-Range" and "If-None-Match" requests as well).
As mentioned previously, it's often preferable to limit on the write side but it's awkward to wrap an http.ResponseWriter since various http functions also check for optional interfaces such as http.Flusher and http.Hijacker. It's much easier to wrap the io.ReadSeeker that ServeContent needs.
For example, something like this perhaps:
func pathFromID(fileID string) string {
// replace with whatever logic you need
return "../../bin/files/test.txt"
}
// or more verbosely you could call this a "limitedReadSeeker"
type lrs struct {
io.ReadSeeker
// This reader must not buffer but just do something simple
// while passing through Read calls to the ReadSeeker
r io.Reader
}
func (r lrs) Read(p []byte) (int, error) {
return r.r.Read(p)
}
func newLRS(r io.ReadSeeker, bucket *ratelimit.Bucket) io.ReadSeeker {
// Here we know/expect that a ratelimit.Reader does nothing
// to the Read calls other than add delays so it won't break
// any io.Seeker calls.
return lrs{r, ratelimit.Reader(r, bucket)}
}
func serveFile(w http.ResponseWriter, req *http.Request) {
fileID := req.URL.Query().Get("fileID")
if len(fileID) == 0 {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
path := pathFromID(fileID)
file, err := os.Open(path)
if err != nil {
http.NotFound(w, req)
return
}
defer file.Close()
fi, err := file.Stat()
if err != nil {
http.Error(w, "blah", 500) // XXX fixme
return
}
const (
rate = 100 << 10
capacity = 100 << 10
)
// Normally we'd prefer to limit the writer but it's awkward to wrap
// an http.ResponseWriter since it may optionally also implement
// http.Flusher, or http.Hijacker.
bucket := ratelimit.NewBucketWithRate(rate, capacity)
lr := newLRS(file, bucket)
http.ServeContent(w, req, path, fi.ModTime(), lr)
}
I'm not seeing the error, but I did notice some issues with the code. For this:
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
You should use the mime package's:
func TypeByExtension(ext string) string
To determine the content type. (if you end up with the empty string default to application/octet-stream)
For:
w.Header().Set("Content-Length", r.Header.Get("Content-Length"))
You need to get the content length from the file itself. By using the request content length, for a GET this basically ends up as a no-op, but for a POST you're sending back the wrong length, which might explain the error you're seeing. After you open the file, do this:
fi, err := file.Stat()
if err != nil {
http.Error(w, err.Error(), 500)
return
}
w.Header().Set("Content-Length", fmt.Sprint(fi.Size()))
One final thing, when you open the file, if there's an error, you don't need to close the file handle. Do it like this instead:
file, err := os.Open(...)
if err != nil {
http.NotFound(w, r)
return
}
defer file.Close()
Is there a way to download a large file using Go that will store the content directly into a file instead of storing it all in memory before writing it to a file? Because the file is so big, storing it all in memory before writing it to a file is going to use up all the memory.
I'll assume you mean download via http (error checks omitted for brevity):
import ("net/http"; "io"; "os")
...
out, err := os.Create("output.txt")
defer out.Close()
...
resp, err := http.Get("http://example.com/")
defer resp.Body.Close()
...
n, err := io.Copy(out, resp.Body)
The http.Response's Body is a Reader, so you can use any functions that take a Reader, to, e.g. read a chunk at a time rather than all at once. In this specific case, io.Copy() does the gruntwork for you.
A more descriptive version of Steve M's answer.
import (
"os"
"net/http"
"io"
)
func downloadFile(filepath string, url string) (err error) {
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Check server response
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("bad status: %s", resp.Status)
}
// Writer the body to file
_, err = io.Copy(out, resp.Body)
if err != nil {
return err
}
return nil
}
The answer selected above using io.Copy is exactly what you need, but if you are interested in additional features like resuming broken downloads, auto-naming files, checksum validation or monitoring progress of multiple downloads, checkout the grab package.
Here is a sample. https://github.com/thbar/golang-playground/blob/master/download-files.go
Also I give u some codes might help you.
code:
func HTTPDownload(uri string) ([]byte, error) {
fmt.Printf("HTTPDownload From: %s.\n", uri)
res, err := http.Get(uri)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
d, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("ReadFile: Size of download: %d\n", len(d))
return d, err
}
func WriteFile(dst string, d []byte) error {
fmt.Printf("WriteFile: Size of download: %d\n", len(d))
err := ioutil.WriteFile(dst, d, 0444)
if err != nil {
log.Fatal(err)
}
return err
}
func DownloadToFile(uri string, dst string) {
fmt.Printf("DownloadToFile From: %s.\n", uri)
if d, err := HTTPDownload(uri); err == nil {
fmt.Printf("downloaded %s.\n", uri)
if WriteFile(dst, d) == nil {
fmt.Printf("saved %s as %s\n", uri, dst)
}
}
}