Go httputil.ReverseProxy not overriding the Host header - go

I'm basically trying to write a reverse proxy server so that when I curl localhost:8080/get it proxies the request to https://nghttp2.org/httpbin/get.
Note: the https://nghttp2.org/httpbin/get service listed above is http/2. But this behavior happens with http/1 as well, such as https://httpbin.org/get.
I'm using httputil.ReverseProxy for this and I'm rewriting the URL while customizing the Host header to not to leak the localhost:8080 to the actual backend.
However, the request still hits the backend with Host: localhost:8080 no matter how many times I set it on the header. Similarly, I used mitmproxy to snoop on the request and it looks like the net/http.Client sets the :authority pseudo-header to localhost:8080
Here's my source code:
package main
import (
"log"
"net/http"
"net/http/httputil"
)
func main() {
proxy := &httputil.ReverseProxy{
Transport: roundTripper(rt),
Director: func(req *http.Request) {
req.URL.Scheme = "https"
req.URL.Host = "nghttp2.org"
req.URL.Path = "/httpbin" + req.URL.Path
req.Header.Set("Host", "nghttp2.org") // <--- I set it here first
},
}
log.Fatal(http.ListenAndServe(":8080", proxy))
}
func rt(req *http.Request) (*http.Response, error) {
log.Printf("request received. url=%s", req.URL)
req.Header.Set("Host", "nghttp2.org") // <--- I set it here as well
defer log.Printf("request complete. url=%s", req.URL)
return http.DefaultTransport.RoundTrip(req)
}
// roundTripper makes func signature a http.RoundTripper
type roundTripper func(*http.Request) (*http.Response, error)
func (f roundTripper) RoundTrip(req *http.Request) (*http.Response, error) { return f(req) }
When I query curl localhost:8080/get the request gets proxied to https://nghttp2.org/httpbin/get. The echoed response shows that clearly my directives setting the Host header didn't do anything:
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "localhost:8080",
"User-Agent": "curl/7.54.0"
},
"origin": "2601:602:9c02:16c2:fca3:aaab:3914:4a71",
"url": "https://localhost:8080/httpbin/get"
}
mitmproxy snooping also clearly shows that the request was made with :authority pseudo-header set to localhost:8080:

From http.Request docs:
// For server requests, Host specifies the host on which the URL
// is sought. Per RFC 7230, section 5.4, this is either the value
// of the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
// To prevent DNS rebinding attacks, server Handlers should
// validate that the Host header has a value for which the
// Handler considers itself authoritative. The included
// ServeMux supports patterns registered to particular host
// names and thus protects its registered Handlers.
//
// For client requests, Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
So the value of URL.Host is only used in case request.Host is empty which is not the case. Setting request.Host should resolve the issue:
req.Host = "nghttp2.org"
Related issue discussed here.

Related

How to set X-Forwarded-For with httputil.ReverseProxy

Why would I want to do this
I have run in to this issue twice.
The first time was with a reverse proxy that lived inside an IPv6 only server network. Client requests come in through NAT46. The source-IP of the request becomes [fixed 96-bit prefix] + [32-bit client IPv4 address]. This means that the reverse proxy could always identify the real client IP. I couldn't find a way to set the X-Forwarded-For header to that address though. I got around it by modifying the backend server.
This time I have a reverse proxy which will run on Google App Engine. Requests hit Google's load balancer first, which adds the X-Forwarded-For header and forwards the request to my app. I want to modify the request a bit and then pass it to a backend server, which I cannot modify. The back-end needs the original client IP, and can accept it via X-Forwarded-For (it's authenticated, don't worry). In this case I want to pass the X-Forwarded-For header from Google's load balencer through unmodified.
The Problem
It seems like there is no way to set X-Forwarded-For to a value that I chose when using httputil.ReverseProxy. If I set it (option 1 below) the client address from the TCP connection will be appended. If if I set it to nil (option 2 below), it is omitted like the documentation suggests, but that's not what I want either.
package main
import (
"log"
"net/http"
"net/http/httputil"
)
func director(req *http.Request) {
// currently running a PHP script that just records headers
host := "bsweb02.bentonvillek12.org"
// who we dial
req.URL.Scheme = "https"
req.URL.Host = host
// host header
req.Host = host
// proof that most headers can be passed through fine
req.Header["Foo"] = []string{"Bar"}
req.Header["X-Forwarded-For"] = []string{"1.2.3.4"} // option 1
//req.Header["X-Forwarded-For"] = nil // option 2
}
func main() {
http.ListenAndServe(":80", &httputil.ReverseProxy{
Director: director,
})
}
Option 1
array(9) {
["Content-Type"]=>
string(0) ""
["Content-Length"]=>
string(1) "0"
["Foo"]=>
string(3) "Bar"
["X-Forwarded-For"]=>
string(18) "1.2.3.4, 127.0.0.1"
["User-Agent"]=>
string(11) "curl/7.81.0"
["Host"]=>
string(26) "bsweb02.bentonvillek12.org"
["Accept-Encoding"]=>
string(4) "gzip"
["Accept"]=>
string(3) "*/*"
["Connection"]=>
string(5) "close"
}
Option 2
array(8) {
["Content-Type"]=>
string(0) ""
["Content-Length"]=>
string(1) "0"
["Foo"]=>
string(3) "Bar"
["User-Agent"]=>
string(11) "curl/7.81.0"
["Host"]=>
string(26) "bsweb02.bentonvillek12.org"
["Accept-Encoding"]=>
string(4) "gzip"
["Accept"]=>
string(3) "*/*"
["Connection"]=>
string(5) "close"
}
I believe you have two options.
1. Implement http.RoundTripper
You implement your own RoundTripper and re-set X-Forwarded-For in there. (demonstration)
type MyRoundTripper struct{}
func (t *MyRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
req.Header["X-Forwarded-For"] = []string{"1.2.3.4"}
return http.DefaultTransport.RoundTrip(req)
}
func main() {
http.ListenAndServe(":80", &httputil.ReverseProxy{
Director: director,
Transport: &MyRoundTripper{},
})
}
When the Transport field isn't set on httputil.ReverseProxy it falls back to http.DefaultTransport, so you can fall back to it too after your custom code.
2. Unset req.RemoteAddr
You reset the original request's RemoteAddr field before invoking the reverse proxy. This field is set by the HTTP server and, when present, triggers the X-Forwarded-For replacement in the reverse proxy implementation. (demonstration)
func main() {
http.ListenAndServe(":80", func(w http.ResponseWriter, r *http.Request) {
r.RemoteAddr = ""
proxy := &httputil.ReverseProxy{ Director: director }
proxy.ServeHTTP(w, r)
})
}
However this behavior relies on an implementation detail, which may or may not change in the future. I recommend using option 1 instead.

How to get full server URL from any endpoint handler in Gin

I'm creating an endpoint using Go's Gin web framework. I need full server URL in my handler function. For example, if server is running on http://localhost:8080 and my endpoint is /foo then I need http://localhost:8080/foo when my handler is called.
If anyone is familiar with Python's fast API, the Request object has a method url_for(<endpoint_name>) which has the exact same functionality: https://stackoverflow.com/a/63682957/5353128
In Go, I've tried accessing context.FullPath() but that only returns my endpoint /foo and not the full URL. Other than this, I can't find appropriate method in docs: https://pkg.go.dev/github.com/gin-gonic/gin#Context
So is this possible via gin.Context object itself or are there other ways as well? I'm completely new to Go.
c.Request.Host+c.Request.URL.Path should work but the scheme has to be determined.
package main
import (
"fmt"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.Default()
r.GET("/foo", func(c *gin.Context) {
fmt.Println("The URL: ", c.Request.Host+c.Request.URL.Path)
})
r.Run(":8080")
}
You can determine scheme which also you may know already. But you can check as follows:
scheme := "http"
if c.Request.TLS != nil {
scheme = "https"
}
If your server is behind the proxy, you can get the scheme by c.Request.Header.Get("X-Forwarded-Proto")
You can get host part localhost:8080 from context.Request.Host and path part /foo from context.Request.URL.String().
package main
import (
"fmt"
"net/http"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.Default()
r.GET("/foo", func(c *gin.Context) {
c.String(http.StatusOK, "bar")
fmt.Println(c.Request.Host+c.Request.URL.String())
})
// Listen and Server in 0.0.0.0:8080
r.Run(":8080")
}
And you can get http protocol version by context.Request.Proto, But it will not determine http or https. you need to get it from your service specifications.

How to send same cookies (CookieJar) in http.Client for different domains

I'm using http.Client for making HTTP requests for some production resource.
This resource has two different domains with the same business-logic
(for example: example.com, instance.com). So ALL cookies for example.com is valid for instance.com and so.
The problem is that I need to send same cookies to two different domains, that is not possible in GoLang.
func (*Jar) Cookies returns cookies for url with a specific domain, so I must call some cookies-preparation function:
func (session *Session) PrepareCookiesForExample() {
example, _ := url.Parse("https://example.com")
session.client.Jar.SetCookies(example, session.client.Jar.Cookies(commu))
}
So I have to call this function in my each request that is pretty uncomfortable and can cause errors (because cookies are not sent) if I forget to call this fuction.
How to send the same cookies for ALL domains by using CookieJar?
First of all, a reminder that restricting cookies to the domains they were set from is an important security feature that should not be bypassed lightly.
Here is an example of how you'd create your own cookie Jar:
package main
import (
"net/http"
"net/url"
)
type SharedCookieJar struct {
CookieSlice []*http.Cookie
}
func (jar *SharedCookieJar) SetCookies(u *url.URL, cookies []*http.Cookie) {
jar.CookieSlice = append(jar.CookieSlice, cookies...)
}
func (jar *SharedCookieJar) Cookies(u *url.URL) []*http.Cookie {
return jar.CookieSlice
}
func main() {
c := http.Client{
Jar:&SharedCookieJar{},
}
c.Get("https://example.com/")
c.Get("https://instance.com/") // will use cookies set by example.com
}
Further reading on interfaces here: https://tour.golang.org/methods/9

How to end to end/integration test a Go app that use a reverse proxy to manage subdomain?

I have a Go app that use Gin gonic and a Nginx reverse proxy that send trafic to another app on domain.com and send all the *.domain.com subdomains traffic directly to my go app.
My Go app then has a middleware that will read the hostname that nginx passes to it from Context and allow my handlers to know what subdomain is being request and return the proper data and cookies for said subdomain.
It's a pretty simple setup and it seems to work fine from my test in postman as all my routes are the same across all my subdomains so this way i can only use one router for all of them instead of one router per subodmain.
Now my big problem come when i'm trying to do end to end testing.
I'm setting up my test like this :
router := initRouter()
w := httptest.NewRecorder()
req, _ := http.NewRequest("POST", "/api/login", bytes.NewBuffer(jsonLogin))
req.Header.Set("Content-Type", "application/json")
router.ServeHTTP(w, req)
assert.Equal(t, 200, w.Code)
with initRouter() returning a gin engine with all my routes and middlewares loaded and the rest as a basic test setup.
Obviously the test will fail as the gin Context won't ever receive a subdomain from context and act as if everything is coming from localhost:8000.
Is there a way to either :
"Mock" a subdomain so that the router think the call is coming from foo.localhost.com instead of localhost
Setup my test suit so that the test request are routed thought nginx.. i'd prefer solution 1 as this would be a mess to setup / maintain.
Edit :
As per the httptest doc i've tried to hard code foo.localhost as the param of the NewRequest but it doesn't really behave as i need it to behave :
NewRequest returns a new incoming server Request, suitable for passing to an http.Handler for testing.
The target is the RFC 7230 "request-target": it may be either a path or an absolute URL. If target is an absolute URL, the host name from the URL is used. Otherwise, "example.com" is used.
When hardcoding http://foo.localhost.com/api/login or foo.localhost.com/api/login as the request target it directly passes it to my router under "foo.localhost.com/api/login" while nginx would just hit the /api/login directly and parse from c.Request.Host
Edit 2:
I'm currently exploring setting the host manually using :
req.Header.Set("Host", "foo.localhost")
The request returned by http.NewRequest isn't suitable for passing directly to ServeHTTP. Use one returned by httptest.NewRequest instead.
Simply set the Host field directly:
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestHelloWorld(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.Host != "foobar" {
t.Errorf("Host is %q, want foobar", r.Host)
}
})
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/api/login", nil)
r.Host = "foobar"
mux.ServeHTTP(w, r)
}

Go webserver - don't cache files using timestamp

I'm running a webserver written in go on an embedded system. The timestamp of index.html may go backwards if someone has downgraded the firmware version. If index.html is older than the previous version, the server sends a http 304 response (not modified), and serves a cached version of the file.
The webserver code is using http.FileServer() and http.ListenAndServe().
The problem can easily reproduced by modifying the timestamp of index.html using the Posix command touch
touch -d"23:59" index.html
reloading the page, then
touch -d"23:58" index.html
reloading this time will give a 304 response on index.html.
Is there a way to prevent timestamp based caching?
Assuming your file server code is like the example in the docs:
http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("/static"))))
You can write a handler that sets the appropriate cache headers to prevent this behaviour by stripping ETag headers and setting Cache-Control: no-cache, private, max-age=0 to prevent caching (both locally and in upstream proxies):
var epoch = time.Unix(0, 0).Format(time.RFC1123)
var noCacheHeaders = map[string]string{
"Expires": epoch,
"Cache-Control": "no-cache, private, max-age=0",
"Pragma": "no-cache",
"X-Accel-Expires": "0",
}
var etagHeaders = []string{
"ETag",
"If-Modified-Since",
"If-Match",
"If-None-Match",
"If-Range",
"If-Unmodified-Since",
}
func NoCache(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
// Delete any ETag headers that may have been set
for _, v := range etagHeaders {
if r.Header.Get(v) != "" {
r.Header.Del(v)
}
}
// Set our NoCache headers
for k, v := range noCacheHeaders {
w.Header().Set(k, v)
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
Use it like so:
http.Handle("/static/", NoCache(http.StripPrefix("/static/", http.FileServer(http.Dir("/static")))))
Note: I originally wrote this at github.com/zenazn/goji/middleware, so you can also just import that, but it's a simple piece of code to write and I wanted to show a full example for posterity!

Resources