How can I test the error for ioutil.ReadAll(rep.Body)? Do I need to split my function in two, one which will make the request, and another one which will read the body and return the bytes and error?
func fetchUrl(URL string) ([]bytes, error) {
resp, err := http.Get(URL)
if err != nil {
return nil, err
}
body, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return nil, err
}
return body, nil
}
Do I need to split my function in two, one which will make the request, and another one which will read the body and return the bytes and error?
The first one is called http.Get and the other one ioutil.ReadAll, so I don't think there's anything to split. You just created a function that uses two other functions together which you should assume are working correctly. You could even simplify your function to make it more obvious:
func fetchURL(URL string) ([]byte, error) {
resp, err := http.Get(URL)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return ioutil.ReadAll(resp.Body)
}
If you want to test anything is your fetchURL function using http.Get and ioutil.ReadAll together. I wouldn't personally bother to test it directly, but if you insist on it, you can overwrite http.DefaultTransport for a single test and provide your own, which returns http.Response with body implementing some error scenarios (e.g. and error during body read).
Here is the sketch idea:
type BrokenTransport struct {
}
func (*BrokenTransport) RoundTrip(*http.Request) (*http.Response, error) {
// Return Response with Body implementing specific error behaviour
}
http.DefaultTransport = &BrokenTransport{}
// http.Get will now use your RoundTripper.
// You should probably restore http.DefaultTransport after the test.
Basically yes, unless you're using net/http/httptest or a similar way to mock your HTTP server when testing.
But the question is: what would you really be testing? That ioutil.ReadAll() detects errors? But I'm sure this was already covered by the test suite of the Go's stdlib.
Hence I'd say that in this particular case you're about to test for the testing's sake. IMO for such trivial cases it's better to concentrate on how the fetched result is further processed.
Related
I'm implementing a http.RoundTripper in Go, and as part of httputil.ReverseProxy implementation.
I need to buffer an incoming request, and repeat it several times, depending on the response I get from the backend. To do this, I use request.Write and http.ReadRequest. (I am actually not sure if this is a good idea, if there are any better ways, I'm interested.)
After deserializing request from []byte with http.ReadRequest and repeat it using the http.DefaultTransport’s roundtripper, I get this printed in my stderr:
2019/08/01 14:35:51 http: proxy error: unsupported protocol scheme ""
So it looks like for some reason I need to set req.URL again after deserializing to make it work.
Here's roughly how my code looks like:
func (s *myServer) RoundTrip(origReq *http.Request) (*http.Response, error) {
var b bytes.Buffer
if err := origReq.Write(&b); err != nil {
return nil, errors.Wrap(err,"failed to buffer request")
}
for retries := 0; retries < s.maxRetries; retries++{
req, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(b.Bytes()))) // probably can be simplified
if err != nil {
return nil, errors.Wrap(err,"failed to un-buffer request")
}
req.URL = origReq.URL // <-- why is this necessary?
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
if needRepeat(resp) {
continue
}
return resp, nil
}
}
ReadRequest
reads a server request. Request.Write writes a client request. See the Request.URL documentation for how the Request.URL is handled differently in client and server requests.
Given that ReadRequest and Request.Write are not inverses of each other,
a better approach is to copy the request body before the loop and create a new request on each iteration using data from the original request and the copied request body.
I'm a little confused when it comes to testing in Go. I've read that abstracting to interfaces should be the ideal way to go in some cases, in other cases I see TestTables. I'm not too sure when to apply either one. For instance, how would one go about testing the function below.
type User{
Name string `json:"name"`
IsMarried bool `json:"isMarried"`
Nicknames []string `json:"nicknames"`
}
func (u *User) Create() (*http.Response, error) {
data, err := json.Marshal(u)
if err != nil {
return nil, err
}
urll := EndpointBase+"/users"
req, err := http.NewRequest(http.MethodPost, urll, bytes.NewReader(data))
if err != nil {
return nil, err
}
resp, err := auth.Session.Client.Do(req)
if err != nil {
return nil, err
}
return resp, nil
}
Abstracting to interfaces and table-driven tests are unrelated concepts that are commonly both used.
You would abstract to interfaces for your dependencies so that you can mock/stub them as needed (in this case, your dependencies are whatever you're calling with HTTP, whatever auth is, and whatever the global EndpointBase is).
Table-driven tests allow you to write multiple test cases more efficiently with less repeated code in your test.
I'd say that unit testing this function won't have much value though, because it's such a thin wrapper around an HTTP call. An integration test would be more useful, in which case abstracting to interfaces wouldn't help with testing (though it could be a good design decision anyway).
I know that I can add headers to each HTTP request manually using
cli := &http.Client{}
req, err := http.NewRequest("GET", "https://myhost", nil)
req.Header.Add("X-Test", "true")
if err != nil {
panic(err)
}
rsp, err := cli.Do(req)
but I want to add this header automatically for each HTTP request in my app.
What is the best way to do it?
I'm aware of three possible solutions to this. In (my) order of preference:
Wrap http.NewRequest with custom code that adds desired headers:
func MyRequest(method, path string, body io.Reader) (*http.Request, error) {
req, err := http.NewRequest(method, path, body)
if err != nil {
return nil, err
}
req.Header.Add("X-Test", "true")
return req, nil
}
This approach has the advantage of being straight-forward, non-magical, and portable. It will work with any third-party software, that adds its own headers, or sets custom transports.
The only case where this won't work is if you depend on a third-party library to create your HTTP requests. I expect this is rare (I don't recall ever running into this in my own experience). And even in such a case, perhaps you can wrap that call instead.
Wrap calls to client.Do to add headers, and possibly any other shared logic.
func MyDo(client *http.Client, req *http.Request) (*http.Response, error) {
req.Header.Add("X-Test", "true")
// Any other common handling of the request
res, err := client.Do(req)
if err != nil {
return nil, err
}
// Any common handling of response
return res, nil
}
This approach is also straight-forward, and has the added advantage (over #1) of making it easy to reduce other boilerplate. This general method can also work very well in conjunction with #1. One possible draw-back is that you must always call your MyDo method directly, meaning you cannot rely on third party software which calls http.Do itself.
Use a custom http.Transport
type myTransport struct{}
func (t *myTransport) RoundTrip(req *http.Request) (*http.Response, error) {
req.Header.Add("X-Test", "true")
return http.DefaultTransport.RoundTrip(req)
}
Then use it like this:
client := &Client{Transport: &myTransport{}}
req := http.NewRequest("GET", "/foo", nil)
res, err := client.Do(req)
This approach has the advantage of working "behind the scenes" with just about any other software, so if you rely on a third-party library to create your http.Request objects, and to call http.Do, this may be your only option.
However, this has the potential disadvantage of being non-obvious, and possibly breaking if you're using any third-party software which also sets a custom transport (without bothering to honor an existing custom transport).
Ultimately, which method you use will depend on what type of portability you need with third-party software. But if that's not a concern, I suggest using the most obvious solution, which, by my estimation, is the order provided above.
It's possible to configure http.Client to use custom transport, which can handle each request in the client (found this implementation in golang.org/x/oauth2 library). This example appends headers to each http request:
type transport struct {
headers map[string]string
base http.RoundTripper
}
func (t *transport) RoundTrip(req *http.Request) (*http.Response, error) {
for k, v := range t.headers {
req.Header.Add(k, v)
}
base := t.base
if base == nil {
base = http.DefaultTransport
}
return base.RoundTrip(req)
}
func main() {
cli := &http.Client{
Transport: &transport{
headers: map[string]string{
"X-Test": "true",
},
},
}
rsp, err := cli.Get("http://localhost:8080")
defer rsp.Body.Close()
if err != nil {
panic(err)
}
}
How can I test func main? Like this:
func main(){
Engine := GetEngine() // returns gin router with handlers atttached
Engine.Run(":8080")
}
It has only 2 lines but I'd like to have them covered.
TestMain' is reserved for test preparation, does that mean testing main was not planned by language creators?
I can move the contents to another function mainReal but it seems to be some over engineering?
How to test gin has started well? Can I launch main in separate goroutine, check reply and stop it?
Thanks.
P.S. Possible duplicate is not precise duplicate because it is dedicated not to testing of func main() itself, but rather ideas to move in outside and so contains different issue and approach.
Solution.
You may test function main() from package main the same way, just do not name it TestMain. I launch it as a separate goroutine, than try to connect to it and perform any request.
I decided to connect to auxilary handler which should respond with a simple json {"status": "ok"}.
In my case:
func TestMainExecution(t *testing.T) {
go main()
resp, err := http.Get("http://127.0.0.1:8080/checkHealth")
if err != nil {
t.Fatalf("Cannot make get: %v\n", err)
}
bodySb, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("Error reading body: %v\n", err)
}
body := string(bodySb)
fmt.Printf("Body: %v\n", body)
var decodedResponse interface{}
err = json.Unmarshal(bodySb, &decodedResponse)
if err != nil {
t.Fatalf("Cannot decode response <%p> from server. Err: %v", bodySb, err)
}
assert.Equal(t, map[string]interface{}{"status": "ok"}, decodedResponse,
"Should return status:ok")
}
I have a function that will be called on every single HTTP GET request. The function reads a file, does some stuff to the contents of that file, and returns a slice of bytes of those contents. That slice of bytes of then written as the response body to the HTTP response writer.
Do I need to use a mutex for any of the steps in this function to prevent locking in the event of multiple HTTP requests trying to read the same file? And if so, would a simple RWMutex locking the reading of the file suffice, since I am not actually writing to it but am creating a copy of its contents?
Here is the function:
// prepareIndex will grab index.html and add a nonce to the script tags for the CSP header compliance.
func prepareIndex(nonce string) []byte {
// Load index.html.
file, err := os.Open("./client/dist/index.html")
if err != nil {
log.Fatal(err)
}
// Convert to goquery document.
doc, err := goquery.NewDocumentFromReader(file)
if err != nil {
fmt.Println(err)
}
// Find all script tags and set nonce.
doc.Find("body > script").SetAttr("nonce", nonce)
// Grab the HTML string.
html, err := doc.Html()
if err != nil {
fmt.Println(err)
}
return []byte(html)
}
I also thought about just loading the file once when main starts, but I was having a problem where only the first request could see the data and the subsequent requests saw nothing. Probably an error in the way I was reading the file. But I actually prefer my current approach because if there are any changes to index.html, I want them to be persisted to the user immediately without having to restart the executable.
Using RWMutex won't protect you from the file being modified by another program. The best option here would be to load your file in a []byte at startup, and instantiate "bytes".Buffer whenever you use goquery.NewDocumentFromReader. In order for the changes to be propagated to the user, you can use fsnotify[1] to detect changes to your file, and update your cached file ([]byte) when necessary (you will need RWMutex for that operation).
For example:
type CachedFile struct {
sync.RWMutex
FileName string
Content []byte
watcher *fsnotify.Watcher
}
func (c *CachedFile) Buffer() *bytes.Buffer {
c.RLock()
defer c.RUnlock()
return bytes.NewBuffer(c.Content)
}
func (c *CachedFile) Load() error {
c.Lock()
content, err := ioutil.ReadAll(c.FileName)
if err != nil {
c.Unlock()
return err
}
c.Content = content
c.Unlock()
}
func (c *CachedFile) Watch() error {
var err error
c.watcher, err = fsnotify.NewWatcher()
if err != nil {
return err
}
go func() {
for ev := range c.watcher.Events {
if ev.Op != fsnotify.Write {
continue
}
err := c.Load()
if err != nil {
log.Printf("loading %q: %s", c.FileName, err)
}
}
}()
err = c.watcher.Add(c.FileName)
if err != nil {
c.watcher.Close()
return err
}
return nil
}
func (c *CachedFile) Close() error {
return c.watcher.Close()
}
[1] https://godoc.org/github.com/fsnotify/fsnotify
If you're modifying the file, you need a mutex. RWMutex should work fine. It looks like you're just reading it, and in that case you should not see any locking behavior or corruption.
The reason you didn't get any data the second time you read from the same file handle is that you're already at the end of the file when you start reading from it the second time. You need to seek back to offset 0 if you want to read the contents again.