Add headers for each HTTP request using client - go

I know that I can add headers to each HTTP request manually using
cli := &http.Client{}
req, err := http.NewRequest("GET", "https://myhost", nil)
req.Header.Add("X-Test", "true")
if err != nil {
panic(err)
}
rsp, err := cli.Do(req)
but I want to add this header automatically for each HTTP request in my app.
What is the best way to do it?

I'm aware of three possible solutions to this. In (my) order of preference:
Wrap http.NewRequest with custom code that adds desired headers:
func MyRequest(method, path string, body io.Reader) (*http.Request, error) {
req, err := http.NewRequest(method, path, body)
if err != nil {
return nil, err
}
req.Header.Add("X-Test", "true")
return req, nil
}
This approach has the advantage of being straight-forward, non-magical, and portable. It will work with any third-party software, that adds its own headers, or sets custom transports.
The only case where this won't work is if you depend on a third-party library to create your HTTP requests. I expect this is rare (I don't recall ever running into this in my own experience). And even in such a case, perhaps you can wrap that call instead.
Wrap calls to client.Do to add headers, and possibly any other shared logic.
func MyDo(client *http.Client, req *http.Request) (*http.Response, error) {
req.Header.Add("X-Test", "true")
// Any other common handling of the request
res, err := client.Do(req)
if err != nil {
return nil, err
}
// Any common handling of response
return res, nil
}
This approach is also straight-forward, and has the added advantage (over #1) of making it easy to reduce other boilerplate. This general method can also work very well in conjunction with #1. One possible draw-back is that you must always call your MyDo method directly, meaning you cannot rely on third party software which calls http.Do itself.
Use a custom http.Transport
type myTransport struct{}
func (t *myTransport) RoundTrip(req *http.Request) (*http.Response, error) {
req.Header.Add("X-Test", "true")
return http.DefaultTransport.RoundTrip(req)
}
Then use it like this:
client := &Client{Transport: &myTransport{}}
req := http.NewRequest("GET", "/foo", nil)
res, err := client.Do(req)
This approach has the advantage of working "behind the scenes" with just about any other software, so if you rely on a third-party library to create your http.Request objects, and to call http.Do, this may be your only option.
However, this has the potential disadvantage of being non-obvious, and possibly breaking if you're using any third-party software which also sets a custom transport (without bothering to honor an existing custom transport).
Ultimately, which method you use will depend on what type of portability you need with third-party software. But if that's not a concern, I suggest using the most obvious solution, which, by my estimation, is the order provided above.

It's possible to configure http.Client to use custom transport, which can handle each request in the client (found this implementation in golang.org/x/oauth2 library). This example appends headers to each http request:
type transport struct {
headers map[string]string
base http.RoundTripper
}
func (t *transport) RoundTrip(req *http.Request) (*http.Response, error) {
for k, v := range t.headers {
req.Header.Add(k, v)
}
base := t.base
if base == nil {
base = http.DefaultTransport
}
return base.RoundTrip(req)
}
func main() {
cli := &http.Client{
Transport: &transport{
headers: map[string]string{
"X-Test": "true",
},
},
}
rsp, err := cli.Get("http://localhost:8080")
defer rsp.Body.Close()
if err != nil {
panic(err)
}
}

Related

What whould be the best way to forward a request by adding headers?

I just started to use Golang and I want to remake my already working NodeJS/TypeScript app in Go.
One endpoint of my API simply adds server-side generated authorization headers and sends a request to a remote API. Basically filling those headers for me by calling my API instead of the remote API.
This is what I am currently writing
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-Id", "---------")
client := &http.Client{}
res, err := client.Do(req)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
var forwardedBody interface{}
json.Unmarshal(body, &forwardedBody)
return ctx.Status(fiber.StatusOK).JSON(forwardedBody)
}
I'd like to know if I am on the right steps, because making a request, parsing the JSON response with ioutil then unmarshall it to send it back seems kind of overboard for the simplicity of what I am trying to achieve ?
Edit: Thank you for the help, this is what I will be going for
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-ID", "---------")
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return ctx.SendStatus(fiber.StatusBadRequest)
}
ctx.Set("Content-Type", "application/json; charset=utf-8")
return ctx.Status(res.StatusCode).SendStream(res.Body)
}
You can use httputil.ReverseProxy. Which takes a base URL and forwards requests to the base URL, concatenating the path.
ReverseProxy is an HTTP Handler that takes an incoming request and sends it to another server, proxying the response back to the client.
http.Handle("/", &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme = "https"
r.URL.Host = "go.dev"
r.Host = r.URL.Host
r.Header.Set("X-Foo", "Bar")
},
})
If you are not serving this from the root path / you can use StripPrefix.
http.HandleFunc("/foo/", http.StripPrefix("/foo/", proxy)
There is also a helper function NewSingleHostReverseProxy, which possibly removes the need to configure the proxy struct yourself. But I think it will be better to set the Host header along with your custom header.
You don't need to attempt to parse the data as JSON. This will be problematic if any of your endpoints don't return JSON, anyway, so just inject the body directly into the response:
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
// Inject the body from the inner response into the actual response so it can be returned
ctx.Response().SetBody(body)
return cx.Status(fiber.StatusOK)

Is there a way to use the same request.Body in multiple handlers without manually write a lot of code or I need to change the way I'm doing this?

This amazing article here: https://www.alexedwards.net/blog/how-to-properly-parse-a-json-request-body explains very well how to write a Golang handler.
I need to use two handlers, one after the other, only if the first one gives error.
Like this:
func main() {
r := chi.NewRouter()
r.Post("/api", MyHandlers)
}
func MyHandlers(w http.ResponseWriter, r *http.Request) {
err := DoSomething(w, r)
if err != nil {
println("OMG! Error!")
DoSomethingWithThisOneInstead(w, r)
}
}
func DoSomething(w http.ResponseWriter, r *http.Request) error {
// here I need to read request's Body
// and I can use io.TeeReader()
// and I can use all the code in the amazing article example
// but I don't want to, because it's a lot of code to maintain
res, err := myLibrary.DoSomething(requestBody)
if err != nil {
return err
}
render.JSON(w, r, res) // go-chi "render" pkg
return nil
}
func DoSomethingWithThisOneInstead(w http.ResponseWriter, r *http.Request) {
// here I need to read request's Body again!
// and I can use all the code in the amazing article example
// but I don't want to, because it's a lot of code to maintain
anotherLibrary.DoSomethingElse.ServeHTTP(w, r)
}
Is there a different method instead of reading twice or more the same request.Body?
Is there a way to avoid writing all that code in the article (which needs to be maintained) and using open source libraries that do it better and are revised by thousands of smarter eyes than mine?
E.G.: Can I use a go-chi method?
Slurp up the bytes and use as needed:
func MyHandlers(w http.ResponseWriter, r *http.Request) {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
// handle error
}
r.Body.Close()
r.Body = ioutil.NopCloser(bytes.NewReader(body))
err := DoSomething(w, r)
if err != nil {
println("OMG! Error!")
r.Body = ioutil.NopCloser(bytes.NewReader(body))
DoSomethingWithThisOneInstead(w, r)
}
}

Repeating an http.Request multiple times inside a reverse proxy

I'm implementing a http.RoundTripper in Go, and as part of httputil.ReverseProxy implementation.
I need to buffer an incoming request, and repeat it several times, depending on the response I get from the backend. To do this, I use request.Write and http.ReadRequest. (I am actually not sure if this is a good idea, if there are any better ways, I'm interested.)
After deserializing request from []byte with http.ReadRequest and repeat it using the http.DefaultTransport’s roundtripper, I get this printed in my stderr:
2019/08/01 14:35:51 http: proxy error: unsupported protocol scheme ""
So it looks like for some reason I need to set req.URL again after deserializing to make it work.
Here's roughly how my code looks like:
func (s *myServer) RoundTrip(origReq *http.Request) (*http.Response, error) {
var b bytes.Buffer
if err := origReq.Write(&b); err != nil {
return nil, errors.Wrap(err,"failed to buffer request")
}
for retries := 0; retries < s.maxRetries; retries++{
req, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(b.Bytes()))) // probably can be simplified
if err != nil {
return nil, errors.Wrap(err,"failed to un-buffer request")
}
req.URL = origReq.URL // <-- why is this necessary?
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
if needRepeat(resp) {
continue
}
return resp, nil
}
}
ReadRequest
reads a server request. Request.Write writes a client request. See the Request.URL documentation for how the Request.URL is handled differently in client and server requests.
Given that ReadRequest and Request.Write are not inverses of each other,
a better approach is to copy the request body before the loop and create a new request on each iteration using data from the original request and the copied request body.

go-micro wrapper as a service

Question is about using go-micro wrapper as a separate service - if anyone knows how to use it properly please let me know. my example - authWrapper, so all api services should be able to use it, it should be discovered via standard service discovery, to make any changes to authWrapper only 1 service should be rebuild (I didn't find a way how to properly pass context.Context from api service to authWrapper via rpc call)
go-micro docs
go-micro wrapper examples
api's code where authWrapper gets called:
func main() {
service := micro.NewService(
micro.Name("go.micro.api.account"),
micro.WrapHandler(AuthWrapper),
)
fmt.Println("service created")
service.Init()
account.RegisterAccountHandler(service.Server(),
&handler.Account{
ProfileServiceClient: profile.NewProfileServiceClient("go.micro.srv.profile", service.Client()),
AuthServiceClient: auth.NewAuthServiceClient("go.micro.srv.auth", service.Client()),
})
if err := service.Run(); err != nil {
log.Fatal(err)
}
}
and authWrapper:
var methodsWithoutAuth = map[string]bool{"Account.Auth": true, "Account.Create": true}
func AuthWrapper(fn server.HandlerFunc) server.HandlerFunc {
return func(ctx context.Context, req server.Request, resp interface{}) error {
fmt.Printf("AuthWrapper, req: %+v", req)
method := req.Method()
fmt.Printf("checking if method allowed, method: %+v", method)
if _, ok := methodsWithoutAuth[method]; ok {
return fn(ctx, req, resp)
}
fmt.Printf("validating token")
authClient := auth.NewAuthServiceClient("go.micro.srv.auth", client.DefaultClient)
meta, ok := metadata.FromContext(ctx)
if !ok {
return errors.New("no auth meta-data found in request")
}
token := meta["Token"]
log.Println("Authenticating with token: ", token)
newCtx := context.WithValue(ctx, "Method", req.Method())
_, err := authClient.ValidateToken(newCtx, &auth.Token{Token: token})
if err != nil {
return err
}
prof, err := authClient.Decode(newCtx, &auth.Token{Token: token})
if err != nil {
return err
}
newCtxWithProf := context.WithValue(newCtx, "Profile", prof.Profile)
return fn(newCtxWithProf, req, resp)
}
}
You can write service wrappers by incorporating the go-micro client in the wrapper code. You find on github many examples how to write a go-micro client, I believe there is one in the greeter example in the go-micro repository.
I use a wrapper to disclose a grpc-interface to a rest-service wrapper by using the client boilerplate.
You can write wrappers to a micro-service for almost any purpose in this way.
Don't worry about the ports the client code needs to address, Consul can handle this just fine for you.

Split function into 2 function for test coverage

How can I test the error for ioutil.ReadAll(rep.Body)? Do I need to split my function in two, one which will make the request, and another one which will read the body and return the bytes and error?
func fetchUrl(URL string) ([]bytes, error) {
resp, err := http.Get(URL)
if err != nil {
return nil, err
}
body, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return nil, err
}
return body, nil
}
Do I need to split my function in two, one which will make the request, and another one which will read the body and return the bytes and error?
The first one is called http.Get and the other one ioutil.ReadAll, so I don't think there's anything to split. You just created a function that uses two other functions together which you should assume are working correctly. You could even simplify your function to make it more obvious:
func fetchURL(URL string) ([]byte, error) {
resp, err := http.Get(URL)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return ioutil.ReadAll(resp.Body)
}
If you want to test anything is your fetchURL function using http.Get and ioutil.ReadAll together. I wouldn't personally bother to test it directly, but if you insist on it, you can overwrite http.DefaultTransport for a single test and provide your own, which returns http.Response with body implementing some error scenarios (e.g. and error during body read).
Here is the sketch idea:
type BrokenTransport struct {
}
func (*BrokenTransport) RoundTrip(*http.Request) (*http.Response, error) {
// Return Response with Body implementing specific error behaviour
}
http.DefaultTransport = &BrokenTransport{}
// http.Get will now use your RoundTripper.
// You should probably restore http.DefaultTransport after the test.
Basically yes, unless you're using net/http/httptest or a similar way to mock your HTTP server when testing.
But the question is: what would you really be testing? That ioutil.ReadAll() detects errors? But I'm sure this was already covered by the test suite of the Go's stdlib.
Hence I'd say that in this particular case you're about to test for the testing's sake. IMO for such trivial cases it's better to concentrate on how the fetched result is further processed.

Resources