Go Nil Dereference Error When Testing HTTP Request - go

I am trying to test a HTTP request in my Go library. Object which makes the call accepts a HTTP client object via dependency injection so in my test I am mocking the HTTP client like this:
func TestMyObject(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, mockJSONResponse)
}))
defer server.Close()
// Make a transport that reroutes all traffic to the example server
transport := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL)
},
}
// Make a http.Client with the transport
httpClient := &http.Client{Transport: transport}
// I am passing the httpClient to my object
}
Here is how the HTTP request is made within my object:
// Make - makes a prepared HTTP request
func (ir *MyObject) Make() *http.Response {
if ir.Err != nil {
return nil
}
ir.resp, ir.Err = ir.Client.Do(ir.req)
runtime.SetFinalizer(ir, func(ir *MyObject) {
ir.resp.Body.Close()
})
ir.logReqRes()
ir.checkErrorResponse()
return ir.resp
}
I am getting nil pointer dereference error though:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
In this function (I am trying to log the response):
// Logs request and response
func (ir *MyObject) logReqRes() {
log.Print("AAAAAAA")
log.Print(ir.resp)
log.Print("AAAAAAA")
if reqInfo, err := httputil.DumpRequest(ir.req, true); err == nil {
log.Print("Logging request:")
log.Print(string(reqInfo))
}
if respInfo, err := httputil.DumpResponse(ir.resp, true); err == nil {
log.Print("Logging response:")
log.Print(string(respInfo))
}
}
As you can see, ir.resp is nil for some reason. Any ideas?

Ok. I found an issue. Better logging of errors actually showed that my mistake was very simple.
I was passing this as a URL to mocked HTTP client:
host/api/v1/tokens/
Which was resulting in this error:
Post host/api/v1/tokens/: unsupported protocol scheme
Apparently the URL must include a proper HTTP scheme (http://) even when you are using a test server.

Related

How to get the decoded token for jwt auth0 and gofiber?

Im using next.js auth0 and a custom golang api backend and I'm
having trouble getting the decoded token on the backend side.
On the frontend side I followed this tutorial -
https://auth0.com/docs/quickstart/webapp/nextjs/01-login
and I managed to send the accessToken to my backend API successfully
on the backend side I followed this tutorial -
https://auth0.com/docs/quickstart/backend/golang/01-authorization
The middleware has successfully verified the token
Example middleware from auth0 implementation
func EnsureValidToken(next http.Handler) http.Handler {
// EnsureValidToken is a middleware that will check the validity of our JWT.
err := godotenv.Load()
if err != nil {
log.Fatal("Error loading .env file")
}
issuerURL, err := url.Parse("https://" + os.Getenv("AUTH0_DOMAIN") + "/")
if err != nil {
log.Fatalf("Failed to parse the issuer url: %v", err)
}
provider := jwks.NewCachingProvider(issuerURL, 5*time.Minute)
jwtValidator, err := validator.New(
provider.KeyFunc,
validator.RS256,
issuerURL.String(),
[]string{os.Getenv("AUTH0_AUDIENCE")},
validator.WithCustomClaims(
func() validator.CustomClaims {
return &CustomClaims{}
},
),
validator.WithAllowedClockSkew(time.Minute),
)
if err != nil {
log.Fatalf("Failed to set up the jwt validator")
}
errorHandler := func(w http.ResponseWriter, r *http.Request, err error) {
log.Printf("Encountered error while validating JWT: %v", err)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusUnauthorized)
w.Write([]byte(`{"message":"Failed to validate JWT."}`))
}
middleware := jwtmiddleware.New(
jwtValidator.ValidateToken,
jwtmiddleware.WithErrorHandler(errorHandler),
)
return middleware.CheckJWT(next)
}
Example token
I'm using https://docs.gofiber.io/ to handle the HTTP methods
Main function
func main() {
// This is to translate the net/http -> fiber http
var ensureValidToken = adaptor.HTTPMiddleware(EnsureValidToken)
app := fiber.New()
app.Use(cors.New())
app.Use(logger.New())
// routes
app.Use(ensureValidToken)
app.Get("/api/books", getAll)
app.Listen(":8080")
}
func getAll(c *fiber.Ctx) error {
token := c.Context().Value(jwtmiddleware.ContextKey{}).(*validator.ValidatedClaims)
// The above code will always panic, I'm assuming that it already stored in the context since it passes the validation
}
Panic example
panic: interface conversion: interface {} is nil, not
*validator.ValidatedClaims
I dig deeper into the auth0 golang implementation, it does store in the context, I think the translation between http.Request to fiber HTTP failed
r = r.Clone(context.WithValue(r.Context(), ContextKey{}, validToken))
Seems like more people have faced the same issue when they used the gofiber adaptor. The way others have solved it was to create their own implementation of HTTPMiddleware middleware adaptor with the only change being that they set the context to the fiber.Ctx.
You can find an the thread on the gofiber/adaptor github page here: https://github.com/gofiber/adaptor/issues/27#issuecomment-1120428400
I got the same panic in the gin framework, I resolved the panic error by changing the code snippet to c.Request.Context().Value() but this is not available in fiber framework. If you want the decoded jwt token either you can get it from the header of the fiber context and decode it appropriately inside the controller, and pass the token you get from the header to the below function and decode.
import (
extract "github.com/golang-jwt/jwt"
"fmt"
)
func Extractor(tokenString string) {
token, _, err := new(extract.Parser).ParseUnverified(tokenString, extract.MapClaims{})
if err != nil {
fmt.Printf("Error %s", err)
}
if claims, ok := token.Claims.(extract.MapClaims); ok {
// obtains claims
subId := fmt.Sprint(claims["sub"])
fmt.Println(subId)
}
}
Implement your logic after this and pass the values you needed to the next handler.

What whould be the best way to forward a request by adding headers?

I just started to use Golang and I want to remake my already working NodeJS/TypeScript app in Go.
One endpoint of my API simply adds server-side generated authorization headers and sends a request to a remote API. Basically filling those headers for me by calling my API instead of the remote API.
This is what I am currently writing
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-Id", "---------")
client := &http.Client{}
res, err := client.Do(req)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
var forwardedBody interface{}
json.Unmarshal(body, &forwardedBody)
return ctx.Status(fiber.StatusOK).JSON(forwardedBody)
}
I'd like to know if I am on the right steps, because making a request, parsing the JSON response with ioutil then unmarshall it to send it back seems kind of overboard for the simplicity of what I am trying to achieve ?
Edit: Thank you for the help, this is what I will be going for
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-ID", "---------")
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return ctx.SendStatus(fiber.StatusBadRequest)
}
ctx.Set("Content-Type", "application/json; charset=utf-8")
return ctx.Status(res.StatusCode).SendStream(res.Body)
}
You can use httputil.ReverseProxy. Which takes a base URL and forwards requests to the base URL, concatenating the path.
ReverseProxy is an HTTP Handler that takes an incoming request and sends it to another server, proxying the response back to the client.
http.Handle("/", &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme = "https"
r.URL.Host = "go.dev"
r.Host = r.URL.Host
r.Header.Set("X-Foo", "Bar")
},
})
If you are not serving this from the root path / you can use StripPrefix.
http.HandleFunc("/foo/", http.StripPrefix("/foo/", proxy)
There is also a helper function NewSingleHostReverseProxy, which possibly removes the need to configure the proxy struct yourself. But I think it will be better to set the Host header along with your custom header.
You don't need to attempt to parse the data as JSON. This will be problematic if any of your endpoints don't return JSON, anyway, so just inject the body directly into the response:
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
// Inject the body from the inner response into the actual response so it can be returned
ctx.Response().SetBody(body)
return cx.Status(fiber.StatusOK)

Repeating an http.Request multiple times inside a reverse proxy

I'm implementing a http.RoundTripper in Go, and as part of httputil.ReverseProxy implementation.
I need to buffer an incoming request, and repeat it several times, depending on the response I get from the backend. To do this, I use request.Write and http.ReadRequest. (I am actually not sure if this is a good idea, if there are any better ways, I'm interested.)
After deserializing request from []byte with http.ReadRequest and repeat it using the http.DefaultTransport’s roundtripper, I get this printed in my stderr:
2019/08/01 14:35:51 http: proxy error: unsupported protocol scheme ""
So it looks like for some reason I need to set req.URL again after deserializing to make it work.
Here's roughly how my code looks like:
func (s *myServer) RoundTrip(origReq *http.Request) (*http.Response, error) {
var b bytes.Buffer
if err := origReq.Write(&b); err != nil {
return nil, errors.Wrap(err,"failed to buffer request")
}
for retries := 0; retries < s.maxRetries; retries++{
req, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(b.Bytes()))) // probably can be simplified
if err != nil {
return nil, errors.Wrap(err,"failed to un-buffer request")
}
req.URL = origReq.URL // <-- why is this necessary?
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
return resp, err
}
if needRepeat(resp) {
continue
}
return resp, nil
}
}
ReadRequest
reads a server request. Request.Write writes a client request. See the Request.URL documentation for how the Request.URL is handled differently in client and server requests.
Given that ReadRequest and Request.Write are not inverses of each other,
a better approach is to copy the request body before the loop and create a new request on each iteration using data from the original request and the copied request body.

http.Request has undefined fields when deploying to appengine

I'm working on a web app and I depend on the following code for authentication (I'm using github.com/dgrijalva/jwt-go package):
func ValidateProtectedPage(protectedPage http.HandlerFunc) http.HandlerFunc {
return http.HandlerFunc(func(res http.ResponseWriter, req *http.Request) {
// If no Auth cookie is set then return a 404 not found
cookie, err := req.Cookie("Auth")
if err != nil {
Forbidden(res)
return
}
// Return a Token using the cookie
token, err := jwt.ParseWithClaims(cookie.Value, &Claims{}, func(token *jwt.Token) (interface{}, error) {
// Make sure token's signature wasn't changed
if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
return nil, fmt.Errorf("Unexpected siging method")
}
return []byte(util.Secret), nil
})
if err != nil {
Forbidden(res)
return
}
// Grab the tokens claims and pass it into the original request
if claims, ok := token.Claims.(*Claims); ok && token.Valid {
ctx := context.WithValue(req.Context(), "Auth", *claims)
protectedPage(res, req.WithContext(ctx))
} else {
Forbidden(res)
}
})}
The issue is, that I'm trying to deploy my app to appengine, and I get the following errors:
planilla-go/controllers/auth.go:45: req.Context undefined (type *http.Request has no field or method Context)
planilla-go/controllers/auth.go:46: req.WithContext undefined (type *http.Request has no field or method WithContext)
From what I read this is due to incompatibilities between appengine's requests and those from go's library, but I couldn't find a workaround yet
Is there a way to wrap or convert the http.Request to use appengine's instead?
Appengine only supports Go 1.6. A bit baffling since it renders my app useless without a major compat overhaul, but that's the current state of the platform

Go can not add accept and content-type headers at the same time

I am trying to write tests for a simple rest application in go. So I write something like this:
func TestMyTestFunc(t *testing.T) {
var w = httptest.NewRecorder()
req, err := http.NewRequest("POST", "/", nil)
if err != nil {
t.Errorf("Error creating request: %s", err.Error())
}
// req.Header.Add("Content-Type", "application/json")
// req.Header.Add("Accept", "application/json")
l.ServeHTTP(w, req) // l is defined somewhere above
// check for w.Code, w.Body
}
This works perfectly fine. Now I would like to add headers. So I add the commented header part and end up with:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
and the error is on this line: l.ServeHTTP(w, req).
Interesting part, that if I set only Content-Type or Accept, the test runs, but if I set both, it fails. What's wrong?
P.S. I also tried to use req.Header.Set, but with no difference.
Here is an stub for my handler:
func (l myHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
// check for validity
if !valid {
http.Error(w, "Invalid Accept", http.StatusBadRequest)
return
}
w.WriteHeader(http.StatusNoContent)
return
}
I think error in your http server/handler implementation. I tried to reproduce it and it worked.
You can see here:
http://play.golang.org/p/n1YBl3OpbN

Resources