golang ReverseProxy not working - proxy

I don't know why this reverse proxy is not working. I've seen several examples and I can't find anything wrong with it.
package main
import (
"log"
"net/url"
"net/http"
"net/http/httputil"
)
func report(r *http.Request){
log.Print("URL: " + r.URL.Path)
log.Print("Scheme: " + r.URL.Scheme)
log.Print("Host: " + r.URL.Host)
//r.URL.Scheme = "http"
//r.URL.Host = "stackoverflow.com"
//r.Header.Set("Host", "stackoverflow.com")
//log.Print("Header Host: " + r.Header.Get("Host"))
}
func main() {
proxy := httputil.NewSingleHostReverseProxy( &url.URL{Scheme:"http",Host:"myrealserver.com"})
proxy.Director = report
// http.Handle("/", proxy)
error := http.ListenAndServe("mylocalhost.com:8080", proxy)
if(error != nil) {
log.Fatal(error)
}
}
It logs:
2014/04/18 21:32:50 URL: /arg/es
2014/04/18 21:32:50 Scheme:
2014/04/18 21:32:50 Host:
2014/04/18 21:32:50 http: proxy error: unsupported protocol scheme ""
2014/04/18 21:32:51 URL: /favicon.ico
2014/04/18 21:32:51 Scheme:
2014/04/18 21:32:51 Host:
2014/04/18 21:32:51 http: proxy error: unsupported protocol scheme ""
If I uncomment the line that redefines the Schema the error message becomes:
2014/04/18 21:38:05 http: proxy error: http: no Host in request URL
If I uncomment the line that redefines the host also, then the target server becomes stackoverflow.com (I mean, it never uses "myrealserver.com").
If I ask for mylocalhost.com:8080/somepath (or even /) then I get a 404 from Stackoverflow, no matter if stackoverflow.com/somepath exists or not. It says:
Couldn't find mylocalhost.com:8080
The Q&A site mylocalhost.com:8080 doesn't seem to exist... yet
It does not translate the Host header automatically.
If then I uncomment the line that sets (and the other one that prints) the Header "Host". Then I can read "stackoverflow.com" in the log, but I still get the same 404 page reporting that I am trying to access "mylocalhost.com".
I'm using go1.2.1 linux/amd64
How is it that I am supposed to make the program work as a proxy?

Thanks to Alex from Golang-nuts, I have the answer now.
This is what Alex said:
Just need to set http.Request.Host [and scheme] in the Director to get this
working: http://play.golang.org/p/I17ZSM6LQb
If you read the source for SingleHostReverseProxy
(http://golang.org/src/pkg/net/http/httputil/reverseproxy.go#L61), it
sets its own Director which you are overriding. So you need to
reimplement what it already does plus the extra Host change.
Anyway, that didn't solve de Header part of the problem: the target server was still receiving "localhost:8080" as the HTTP Host name, so I did it without the ReverseProxy package, just with http and a RoundTripper, plus a helper function that copies all the headers:
package main
import (
"flag"
"fmt"
"os"
"log"
"net/http"
"io/ioutil"
)
var target *string
func main() {
target = flag.String("target", "http://stackoverflow.com", "target URL for reverse proxy")
flag.Parse()
http.HandleFunc("/", report)
log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}
func report(w http.ResponseWriter, r *http.Request){
uri := *target+r.RequestURI
fmt.Println(r.Method + ": " + uri)
if r.Method == "POST" {
body, err := ioutil.ReadAll(r.Body)
fatal(err)
fmt.Printf("Body: %v\n", string(body));
}
rr, err := http.NewRequest(r.Method, uri, r.Body)
fatal(err)
copyHeader(r.Header, &rr.Header)
// Create a client and query the target
var transport http.Transport
resp, err := transport.RoundTrip(rr)
fatal(err)
fmt.Printf("Resp-Headers: %v\n", resp.Header);
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fatal(err)
dH := w.Header()
copyHeader(resp.Header, &dH)
dH.Add("Requested-Host", rr.Host)
w.Write(body)
}
func fatal(err error) {
if err != nil {
log.Fatal(err)
os.Exit(1)
}
}
func copyHeader(source http.Header, dest *http.Header){
for n, v := range source {
for _, vv := range v {
dest.Add(n, vv)
}
}
}
Now I'm able to see StackOverflow or any other site how it's supposed to be.
I'm still working on POST calls, though, so this is a work in progress.

A little late to the party, but ReverseProxy isn't broken, it's just a little confusing because it doesn't work how you'd expect (at the least, I expected it to work the way you did, so that makes two of us).
From the docs:
// For server requests Host specifies the host on which the
// URL is sought. Per RFC 2616, this is either the value of
// the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
//
// For client requests Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
Since under the hood ReverseProxy is using this Request to make a client request (after ReverseProxy.Director optionally modifies it), if the Host is set it will override the Host header. This will always be set, because as the first part of the doc comment states "For server requests, Host specifies the host on which the URL is sought".
So in addition to Sebastián's answer, you also need to set req.Host. For example, to proxy to example.com:
proxy := ReverseProxy{
Director: func(req *http.Request) {
req.URL.Scheme = "http"
req.URL.Host = "example.com"
req.Host = "example.com"
}
}
Alternatively you can set req.Host to "" and it will use the value of req.URL.Host.
I figured this out by reading: https://github.com/golang/go/issues/14413

Related

Issues creating an HTTPS server to serve binaries and other files

What I want to achieve:
An HTTPS server designed specifically to serve binaries to around 1000 devices, sometimes in the same time (clients will fetch the binaries via wget, curl, browser download, etc).
Key functionality features:
client won't be able to download the binary without a certificate
server will allow the client directory browsing/download via browser(if client has certificate)
server is optimized for stability and security, then for speed
server must use high security ciphers and TLS1.2
What I managed to achieve
package main
import (
"log"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
http.ServeFile(w, req, "/")
})
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(http.ListenAndServeTLS(":9900", "cert.crt", "priv.key", http.FileServer(http.Dir("/"))))
}
Now, this works fine although it doesn't check all the features and its not very flexible and somehow I wanted to make it more future proof, as I wish to both learn by creating this project and also expand on it in the future as I'm interested in learning more about servers.
After a bit of research I found several code pieces on GitHub and in tutorials, which led me to put together the following piece of code:
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
http.ServeFile(w, req, "/")
})
cfg := &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
srv := &http.Server{
Addr: ":9900",
Handler: mux,
TLSConfig: cfg,
TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
}
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(srv.ListenAndServeTLS("cert.crt", "priv.key"), http.FileServer(http.Dir("/")))
}
Problem is that when the server starts and I connect to it via browser, I'm presented with the root directory but every time I click on a folder the URL increments the address but the page just refreshes in the "/" directory.
To be exact:
I connect initially to the server and I'm presented the root directory , I'm shown Dir1, Dir2, Dir3
I click Dir1
The URL modifies from https://localhost:9900 to https://localhost:9900/Dir1
But I'm still in the root directory
From what I'm able to see...I think I'm creating a loop somewhere.
If anyone knows what I need to do to make this functional, help would be much appreciated.
NOTE
The above behavior is on Firefox, on Chrome I get one of the 2 errors in the server error log, depending on changes that I make:
2019/09/29 19:59:37 http: TLS handshake error from [::1]:53287: EOF
2019/09/29 19:15:59 http: TLS handshake error from [::1]:50457: tls: client doesn't support selected certificate
There are several examples elsewhere on how to do this (as has been commented).
Here's a worked example in which the fileserver handler is an enhanced version of that in the standard library that supports more cache headers and locally-compressed files. The standard fileserver handler can be dropped in here instead if that's what you need.
package main
import (
"crypto/tls"
"flag"
"fmt"
"log"
"net/http"
"time"
"github.com/rickb777/servefiles/v3"
)
var path = flag.String("path", "..", "directory for the files tp be served")
var cert = flag.String("cert", "", "file containing the certificate (optional)")
var key = flag.String("key", "", "file containing the private key (optional)")
var port = flag.Int("port", 8080, "TCP port to listen on")
var maxAge = flag.String("maxage", "", "Maximum age of assets sent in response headers - causes client caching")
var verbose = flag.Bool("v", false, "Enable verbose messages")
func main() {
flag.Parse()
if *verbose {
servefiles.Debugf = log.Printf
}
if (*cert != "" && *key == "") ||
(*cert == "" && *key != "") {
log.Fatal("Both certificate file (-cert) and private key file (-key) are required.")
}
h := servefiles.NewAssetHandler(*path)
if *maxAge != "" {
d, err := time.ParseDuration(*maxAge)
log.Printf("MaxAge: %s %v\n", d, err)
h = h.WithMaxAge(d)
}
srv := &http.Server{
Addr: fmt.Sprintf(":%d", *port),
Handler: h,
}
if *cert != "" {
srv.TLSConfig = &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
log.Printf("Access the server via: https://localhost:%d/", *port)
log.Fatal(srv.ListenAndServeTLS(*cert, *key))
} else {
log.Printf("Access the server via: http://localhost:%d/", *port)
log.Fatal(srv.ListenAndServe())
}
}
Source code https://github.com/rickb777/servefiles/blob/master/v3/webserver/example.go

Make httputil.ReverseProxy Foward Response Stream Correctly

I have reverse proxies in my main web-server that are dedicated to a certain micro-service and handle forward requests to their appropriate micro-services.
func newTrimPrefixReverseProxy(target *url.URL, prefix string) *httputil.ReverseProxy {
director := func(req *http.Request) {
// ... trims prefix from request path and prepends the path of the target url
}
return &httputil.ReverseProxy{Director: director}
}
This has worked perfectly for pure JSON responses, but I have ran into issues recently when trying to serve content (stream responses) through the reverse proxy. The means for serving the content is irrelevant, the (video) content is served as intended when the service is accessed directly and not through the reverse proxy.
Serving the content:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
http.ServeContent(w, r, "video.mp4", time.Now().Add(time.Hour*24*365*12*9*-1), videoReadSeeker)
})
Again, the videoReadSeeker and how the content is served is not the issue, the issue is having my response relayed as intended to the requester through the reverse proxy; when accessing the service directly, the video shows up and I can scrub it to my heart's content.
Note that the response for data the content is received (http status, headers), but the content stream in the response body is not.
How can I make sure that the reverse proxy handles streamed responses as intended for the content?
Do you get the same results when using:
package main
import (
"log"
"net/http"
"net/http/httputil"
"net/url"
)
func main() {
u, err := url.Parse("http://localhost:8080/asdfasdf")
if err != nil {
log.Fatal("url.Parse: %v", err)
}
proxy := httputil.NewSingleHostReverseProxy(u)
log.Printf("Listening at :8081")
if err := http.ListenAndServe(":8081", proxy); err != nil {
log.Fatal("ListenAndServe: %v", err)
}
}
Ultimately these are the same implementation under the hood, but the director provided here ensures that some of the expected headers exist that you will need for some proxy features to function as expected.

How to use http.Get request inside handler func

How to make http.Get request inside of handler func?
For example, this simple code "should" return blank page in localhost:8080 browser but it go nuts. What I have missed in school?
package main
import "net/http"
func index(w http.ResponseWriter, r *http.Request) {
_, err := http.Get("www.google.com")
if err != nil {
panic(err)
}
}
func main() {
http.HandleFunc("/", index)
http.ListenAndServe(":8080", nil)
}
The problem is that you should use a protocol (e.g. https://) in the Get function:
_, err := http.Get("https://www.google.com")
The error in you original code is Get www.google.com: unsupported protocol scheme "".
http.Get() expects a URL, and www.google.com is not a URL; a URL begins with a scheme. Even though it is common to type "www.google.com" into a browser, it's still not a full URL; friendly browsers automatically prepend "http://" or "https://" before issuing the request. http.Get() isn't going to do that; it expects a well-formed URL to begin with.

Go & Socket.io HTTP + WSS on one port with CORS?

Brand new to Go.. Still obviously learning the syntax and the basics.. But I do have a specific goal in mind..
I'm trying to just get a simple server up on :8080 that can respond to both HTTP and socket.io (via /socket.io/ url), specificaly with CORS.
My code:
package main
import (
"log"
"net/http"
"github.com/rs/cors"
"github.com/googollee/go-socket.io"
)
func SayHelloWorld(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello, World!"))
}
func main() {
c := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
AllowCredentials: true,
})
server, err := socketio.NewServer(nil)
if err != nil {
log.Fatal(err)
}
server.On("connection", func(so socketio.Socket) {
log.Println("on connection")
so.Join("chat")
so.On("chat message", func(msg string) {
log.Println("emit:", so.Emit("chat message", msg))
so.BroadcastTo("chat", "chat message", msg)
})
so.On("disconnection", func() {
log.Println("on disconnect")
})
})
server.On("error", func(so socketio.Socket, err error) {
log.Println("error:", err)
})
http.Handle("/socket.io/", c.Handler(server))
http.HandleFunc("/", SayHelloWorld)
log.Println("Serving at localhost:8080...")
log.Fatal(http.ListenAndServe(":8080", nil))
}
On the client side I'm still seeing:
WebSocket connection to 'wss://api.domain.com/socket.io/?EIO=3&transport=websocket&sid=xNWd9aZvwDnZOrXkOBaC' failed: WebSocket is closed before the connection is established.
(index):1 XMLHttpRequest cannot load https://api.domain.com/socket.io/?EIO=3&transport=polling&t=1420662449235-3932&sid=xNWd9aZvwDnZOrXkOBaC. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://fiddle.jshell.net' is therefore not allowed access.
EDIT #1:
So I've been banging my head away trying to understand why I can't connect.. Came across an even more confusing piece of the puzzle?
https://gist.github.com/acoyfellow/167b055da85248c94fc4
The above gist is the code of my golang server + the browser code used to connect.. This code will send 30 HTTP GET requests per second to the backend, without connecting, upgrading, or giving any errors (client or server side).. it essentially DDOS's my own backend?
Someone, please someone tell me I'm doing something stupid.. This is quite the pickle :P
EDIT #2:
I can stop the "DDOS" by simply adjusting the trailing / on the URL of the socket.io endpoint in Go.. So: mux.Handle("/socket.io", server) to mux.Handle("/socket.io/", server) will now produce error messages and connection attempts with error responses of:
WebSocket connection to 'wss://api.domain.com/socket.io/?EIO=3&transport=websocket&sid=0TzmTM_QtF1TaS4exiwF' failed: Error during WebSocket handshake: Unexpected response code: 400 socket.io-1.2.1.js:2
GET https://api.domain.com/socket.io/?EIO=3&transport=polling&t=1420743204485-62&sid=0TzmTM_QtF1TaS4exiwF 400 (Bad Request)
So I gave up using googoolee's Socket.io implementation and went with gorilla's.
I checked out their examples: https://github.com/gorilla/websocket/tree/master/examples/chat
Checked out their docs: http://www.gorillatoolkit.org/pkg/websocket
-- Under Origin Considerations I found:
An application can allow connections from any origin by specifying a function that always returns true:
var upgrader = websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool { return true },
}
I added this CheckOrigin function to the conn.go file in their example, and was able to get a CORS socket server talking to a browser.
As a first adventure into Golang, this was frustrating and fun.. +1 to anyone else learning
Don't you mean http + ws or https + wss. If you remove a s from wss, you should be able to connect.
If you want tls for web socket (wss), then you need to http.ListenAndServeTLS.
It appears that CORS does not apply to WebSockets. Per this related question "With WebSocket, there is an "origin" header, which browser MUST fill with the origin of the HTML containing the JS that opens the WS connection."
As stated here:
Cross origin websockets with Golang
How about in your SayHelloWorld func, adding something like:
w.Header().Set("Access-Control-Allow-Origin", "*")
Or, possibly better:
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
}
I get the similar problerm with normal ajax call. It require more work in both front-end and backend. I belive most popular front-end libs liek JQuery or AngularJS handle these very well.
I see you're using the https://github.com/rs/cors package but you don't include the usage of that package, here is the implement with only Go std package:
type CrossOriginServer struct {}
func (s *CrossOriginServer) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
// you may need to add some more headers here
allowHeaders := "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization"
if origin := req.Header.Get("Origin"); validOrigin(origin) {
rw.Header().Set("Access-Control-Allow-Origin", origin)
rw.Header().Set("Access-Control-Allow-Methods", "POST, PUT, PATCH, GET, DELETE")
rw.Header().Set("Access-Control-Allow-Headers", allowHeaders)
}
if req.Method == "OPTIONS" {
return
}
// if you want, you can use gorilla/mux or any routing package here
mux := http.NewServeMux()
mux.Handle("/socket.io/", c.Handler(server))
mux.HandleFunc("/", SayHelloWorld)
mux.ServeHTTP(rw, req)
}
func validOrigin(origin string) bool {
allowOrigin := []string{
"http://localhost:8081",
"http://example.com"
}
for _, v := range allowOrigin {
if origin == v {
return true
}
}
return false
}
func main() {
// do you stuff
// ...
// ...
http.ListenAndServe(":8080", &CrossOriginServer{})
}

How to get URL in http.Request

I built an HTTP server. I am using the code below to get the request URL, but it does not get full URL.
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Printf("Req: %s %s", r.URL.Host, r.URL.Path)
}
I only get "Req: / " and "Req: /favicon.ico".
I want to get full client request URL as "1.2.3.4/" or "1.2.3.4/favicon.ico".
Thanks.
From the documentation of net/http package:
type Request struct {
...
// The host on which the URL is sought.
// Per RFC 2616, this is either the value of the Host: header
// or the host name given in the URL itself.
// It may be of the form "host:port".
Host string
...
}
Modified version of your code:
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Printf("Req: %s %s\n", r.Host, r.URL.Path)
}
Example output:
Req: localhost:8888 /
I use req.URL.RequestURI() to get the full url.
From net/http/requests.go :
// RequestURI is the unmodified Request-URI of the
// Request-Line (RFC 2616, Section 5.1) as sent by the client
// to a server. Usually the URL field should be used instead.
// It is an error to set this field in an HTTP client request.
RequestURI string
If you detect that you are dealing with a relative URL (r.URL.IsAbs() == false), you sill have access to r.Host (see http.Request), the Host itself.
Concatenating the two would give you the full URL.
Generally, you see the reverse (extracting Host from an URL), as in gorilla/reverse/matchers.go
// getHost tries its best to return the request host.
func getHost(r *http.Request) string {
if r.URL.IsAbs() {
host := r.Host
// Slice off any port information.
if i := strings.Index(host, ":"); i != -1 {
host = host[:i]
}
return host
}
return r.URL.Host
}

Resources