Issues creating an HTTPS server to serve binaries and other files - go

What I want to achieve:
An HTTPS server designed specifically to serve binaries to around 1000 devices, sometimes in the same time (clients will fetch the binaries via wget, curl, browser download, etc).
Key functionality features:
client won't be able to download the binary without a certificate
server will allow the client directory browsing/download via browser(if client has certificate)
server is optimized for stability and security, then for speed
server must use high security ciphers and TLS1.2
What I managed to achieve
package main
import (
"log"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
http.ServeFile(w, req, "/")
})
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(http.ListenAndServeTLS(":9900", "cert.crt", "priv.key", http.FileServer(http.Dir("/"))))
}
Now, this works fine although it doesn't check all the features and its not very flexible and somehow I wanted to make it more future proof, as I wish to both learn by creating this project and also expand on it in the future as I'm interested in learning more about servers.
After a bit of research I found several code pieces on GitHub and in tutorials, which led me to put together the following piece of code:
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
http.ServeFile(w, req, "/")
})
cfg := &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
srv := &http.Server{
Addr: ":9900",
Handler: mux,
TLSConfig: cfg,
TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
}
log.Printf("Server running\nAccess the server via: https://localhost:9900/")
log.Fatal(srv.ListenAndServeTLS("cert.crt", "priv.key"), http.FileServer(http.Dir("/")))
}
Problem is that when the server starts and I connect to it via browser, I'm presented with the root directory but every time I click on a folder the URL increments the address but the page just refreshes in the "/" directory.
To be exact:
I connect initially to the server and I'm presented the root directory , I'm shown Dir1, Dir2, Dir3
I click Dir1
The URL modifies from https://localhost:9900 to https://localhost:9900/Dir1
But I'm still in the root directory
From what I'm able to see...I think I'm creating a loop somewhere.
If anyone knows what I need to do to make this functional, help would be much appreciated.
NOTE
The above behavior is on Firefox, on Chrome I get one of the 2 errors in the server error log, depending on changes that I make:
2019/09/29 19:59:37 http: TLS handshake error from [::1]:53287: EOF
2019/09/29 19:15:59 http: TLS handshake error from [::1]:50457: tls: client doesn't support selected certificate

There are several examples elsewhere on how to do this (as has been commented).
Here's a worked example in which the fileserver handler is an enhanced version of that in the standard library that supports more cache headers and locally-compressed files. The standard fileserver handler can be dropped in here instead if that's what you need.
package main
import (
"crypto/tls"
"flag"
"fmt"
"log"
"net/http"
"time"
"github.com/rickb777/servefiles/v3"
)
var path = flag.String("path", "..", "directory for the files tp be served")
var cert = flag.String("cert", "", "file containing the certificate (optional)")
var key = flag.String("key", "", "file containing the private key (optional)")
var port = flag.Int("port", 8080, "TCP port to listen on")
var maxAge = flag.String("maxage", "", "Maximum age of assets sent in response headers - causes client caching")
var verbose = flag.Bool("v", false, "Enable verbose messages")
func main() {
flag.Parse()
if *verbose {
servefiles.Debugf = log.Printf
}
if (*cert != "" && *key == "") ||
(*cert == "" && *key != "") {
log.Fatal("Both certificate file (-cert) and private key file (-key) are required.")
}
h := servefiles.NewAssetHandler(*path)
if *maxAge != "" {
d, err := time.ParseDuration(*maxAge)
log.Printf("MaxAge: %s %v\n", d, err)
h = h.WithMaxAge(d)
}
srv := &http.Server{
Addr: fmt.Sprintf(":%d", *port),
Handler: h,
}
if *cert != "" {
srv.TLSConfig = &tls.Config{
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_256_CBC_SHA,
tls.TLS_AES_256_GCM_SHA384,
},
}
log.Printf("Access the server via: https://localhost:%d/", *port)
log.Fatal(srv.ListenAndServeTLS(*cert, *key))
} else {
log.Printf("Access the server via: http://localhost:%d/", *port)
log.Fatal(srv.ListenAndServe())
}
}
Source code https://github.com/rickb777/servefiles/blob/master/v3/webserver/example.go

Related

Go Gorilla session not working when hosting on Apache Web Server

Here is the Go code for my sessionized login form
package main
import (
"fmt"
"log"
"net/http"
"github.com/gorilla/context"
"github.com/gorilla/sessions"
)
var store = sessions.NewCookieStore([]byte("super-secret"))
func loginAuthHandler(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
username := r.FormValue("username")
password := r.FormValue("password")
if password == "welcome" && username == "guest" {
session, _ := store.Get(r, "session")
session.Values["authenticated"] = true
err := session.Save(r, w)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
http.Redirect(w, r, "/secret", http.StatusFound)
} else {
fmt.Fprintf(w, "Wrong Login!")
}
}
func secret(w http.ResponseWriter, r *http.Request) {
session, _ := store.Get(r, "session")
if auth, ok := session.Values["authenticated"].(bool); !ok || !auth {
http.Error(w, "Forbidden", http.StatusForbidden)
return
}
fmt.Fprintf(w, "The cake is a lie!")
}
func main() {
store.Options = &sessions.Options{
Domain: "localhost",
Path: "/",
MaxAge: 3600 * 8,
HttpOnly: true,
}
http.HandleFunc("/secret", secret)
http.HandleFunc("/loginauth", loginAuthHandler)
http.Handle("/", http.FileServer(http.Dir("public")))
log.Fatal(http.ListenAndServe(":3003", context.ClearHandler(http.DefaultServeMux)))
}
For some reason this works when I host the server locally but when I try hosting it on linux Apache Web Server on my Digital Ocean Droplet it stops working. The website even redirects correctly to the secret endpoint but it shows the Forbidden message because the cookie never gets set. Is there some kind of difference that Hosting on Apache makes that is causing this?
Here is the code I used to setup the proxy on my server for Apache, other than that I've kept the server mostly default.
#go
ProxyPass /go http://localhost:3003
ProxyPassReverse /go http://localhost:3003
You are setting store.Options.Domain to localhost. The Domain is used when building the session cookie; meaning that the cookies domain attribute will be set to localhost.
As you are hosting your server behind an Apache reverse proxy on a cloud server you will not be accessing it with a localhost URL (e.g. http://localhost/go). This means that the domain within your URL will not be localhost and the cookie will effectively be ignored. See this question for more info.
The quick fix is to leave the Domain unset e.g.
store.Options = &sessions.Options{
Path: "/",
MaxAge: 3600 * 8,
HttpOnly: true,
}
Note: If you do this the cookie will not be available on subdomains (but that si probably not an issue whilst testing).

wildcard certificate support using Autocert (golang)

implementing a https go server with wildcard certificate support.
package main
import (
"crypto/tls"
"log"
"net/http"
"golang.org/x/crypto/acme/autocert"
)
func main() {
certManager := autocert.Manager{
Prompt: autocert.AcceptTOS,
HostPolicy: autocert.HostWhitelist("example.com"), //Your domain here
Cache: autocert.DirCache("certs"), //Folder for storing certificates
}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello world"))
})
server := &http.Server{
Addr: ":https",
TLSConfig: &tls.Config{
GetCertificate: certManager.GetCertificate,
},
}
go http.ListenAndServe(":http", certManager.HTTPHandler(nil))
log.Fatal(server.ListenAndServeTLS("", "")) //Key and cert are coming from Let's Encrypt
}
couldn't figure out how to add a wildcard pattern to the hostwhitelist.
need support for "*.example.com"
The HostWhitelist doesn't support wildcards, but because a HostPolicy is merely a function, you can implement your own HostPolicy, using e.g. a regular expression:
var (
allowedHosts = regexp.MustCompile(`^[^.]+\.example\.com$`)
errPolicyMismatch = errors.New("the host did not match the allowed hosts")
)
func CustomHostPolicy(_ context.Context, host string) error {
if matches := allowedHosts.MatchString(host); !matches {
return errPolicyMismatch
}
return nil
}
See demo on https://go.dev/play/p/8gGIpnl1NLs

How to override default DNS

I've been working on finding a way to try and override the default DNS server for a Go program for a while but still with no success sadly.
The current solution which I had thought would work was:
package main
import (
"context"
"fmt"
"net"
"time"
)
func main() {
DNS := "1.1.1.1"
net.DefaultResolver = &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
d := net.Dialer{
Timeout: time.Millisecond * time.Duration(3000),
}
return d.DialContext(ctx, "udp", fmt.Sprintf("%s:53", DNS))
},
}
resp, err := net.LookupHost("tsdfsdf.com")
if err != nil {
fmt.Printf(err.Error())
}
fmt.Printf("%+v", resp)
}
But the response from this is:
lookup tsdfsdf.com on 192.168.0.1:53: no such host[]
Which is sadly my default DNS server set in my reslov.conf
I have tried forcing to use the Go Resolver by setting export GODEBUG=netdns=go
The long term solution is to be able to over the default resolver for the HTTP Client which would, in turn be consumed by some AWS SDK stuff.
Has any faced this or knows how I can get around this?
FYI, I know "tsdfsdf.com" is not a real domain I'm just using it to spit a No such host error to see what DNS it asked.

Configuring proxy settings for goproxy

I'm trying to use goproxy as an alternative to fiddler on mac OSX,
I wrote the following code, and succeeded to run it:
package main
import (
"flag"
"log"
"net/http"
"gopkg.in/elazarl/goproxy.v1"
)
func main() {
verbose := flag.Bool("v", false, "should every proxy request be logged
to stdout")
addr := flag.String("addr", ":8080", "proxy listen address")
proxy := goproxy.NewProxyHttpServer()
proxy.Verbose = *verbose
proxy.OnResponse().DoFunc(func(resp *http.Response, ctx
*goproxy.ProxyCtx) *http.Response {
contentType := resp.Header.Get("Content-Type")
if contentType == "application/javascript" || contentType == "application/x-javascript" {
// Do something...
}
return resp
})
log.Fatal(http.ListenAndServe(*addr, proxy))
}
According to goproxy documentation I need to configure web proxy & secure web proxy:
I have put multiple breakpoints and run the code in debug, tried to access a website but I can't get it to stop on the breakpoint..
I think I'm missing something in the proxy settings maybe.

golang ReverseProxy not working

I don't know why this reverse proxy is not working. I've seen several examples and I can't find anything wrong with it.
package main
import (
"log"
"net/url"
"net/http"
"net/http/httputil"
)
func report(r *http.Request){
log.Print("URL: " + r.URL.Path)
log.Print("Scheme: " + r.URL.Scheme)
log.Print("Host: " + r.URL.Host)
//r.URL.Scheme = "http"
//r.URL.Host = "stackoverflow.com"
//r.Header.Set("Host", "stackoverflow.com")
//log.Print("Header Host: " + r.Header.Get("Host"))
}
func main() {
proxy := httputil.NewSingleHostReverseProxy( &url.URL{Scheme:"http",Host:"myrealserver.com"})
proxy.Director = report
// http.Handle("/", proxy)
error := http.ListenAndServe("mylocalhost.com:8080", proxy)
if(error != nil) {
log.Fatal(error)
}
}
It logs:
2014/04/18 21:32:50 URL: /arg/es
2014/04/18 21:32:50 Scheme:
2014/04/18 21:32:50 Host:
2014/04/18 21:32:50 http: proxy error: unsupported protocol scheme ""
2014/04/18 21:32:51 URL: /favicon.ico
2014/04/18 21:32:51 Scheme:
2014/04/18 21:32:51 Host:
2014/04/18 21:32:51 http: proxy error: unsupported protocol scheme ""
If I uncomment the line that redefines the Schema the error message becomes:
2014/04/18 21:38:05 http: proxy error: http: no Host in request URL
If I uncomment the line that redefines the host also, then the target server becomes stackoverflow.com (I mean, it never uses "myrealserver.com").
If I ask for mylocalhost.com:8080/somepath (or even /) then I get a 404 from Stackoverflow, no matter if stackoverflow.com/somepath exists or not. It says:
Couldn't find mylocalhost.com:8080
The Q&A site mylocalhost.com:8080 doesn't seem to exist... yet
It does not translate the Host header automatically.
If then I uncomment the line that sets (and the other one that prints) the Header "Host". Then I can read "stackoverflow.com" in the log, but I still get the same 404 page reporting that I am trying to access "mylocalhost.com".
I'm using go1.2.1 linux/amd64
How is it that I am supposed to make the program work as a proxy?
Thanks to Alex from Golang-nuts, I have the answer now.
This is what Alex said:
Just need to set http.Request.Host [and scheme] in the Director to get this
working: http://play.golang.org/p/I17ZSM6LQb
If you read the source for SingleHostReverseProxy
(http://golang.org/src/pkg/net/http/httputil/reverseproxy.go#L61), it
sets its own Director which you are overriding. So you need to
reimplement what it already does plus the extra Host change.
Anyway, that didn't solve de Header part of the problem: the target server was still receiving "localhost:8080" as the HTTP Host name, so I did it without the ReverseProxy package, just with http and a RoundTripper, plus a helper function that copies all the headers:
package main
import (
"flag"
"fmt"
"os"
"log"
"net/http"
"io/ioutil"
)
var target *string
func main() {
target = flag.String("target", "http://stackoverflow.com", "target URL for reverse proxy")
flag.Parse()
http.HandleFunc("/", report)
log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}
func report(w http.ResponseWriter, r *http.Request){
uri := *target+r.RequestURI
fmt.Println(r.Method + ": " + uri)
if r.Method == "POST" {
body, err := ioutil.ReadAll(r.Body)
fatal(err)
fmt.Printf("Body: %v\n", string(body));
}
rr, err := http.NewRequest(r.Method, uri, r.Body)
fatal(err)
copyHeader(r.Header, &rr.Header)
// Create a client and query the target
var transport http.Transport
resp, err := transport.RoundTrip(rr)
fatal(err)
fmt.Printf("Resp-Headers: %v\n", resp.Header);
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fatal(err)
dH := w.Header()
copyHeader(resp.Header, &dH)
dH.Add("Requested-Host", rr.Host)
w.Write(body)
}
func fatal(err error) {
if err != nil {
log.Fatal(err)
os.Exit(1)
}
}
func copyHeader(source http.Header, dest *http.Header){
for n, v := range source {
for _, vv := range v {
dest.Add(n, vv)
}
}
}
Now I'm able to see StackOverflow or any other site how it's supposed to be.
I'm still working on POST calls, though, so this is a work in progress.
A little late to the party, but ReverseProxy isn't broken, it's just a little confusing because it doesn't work how you'd expect (at the least, I expected it to work the way you did, so that makes two of us).
From the docs:
// For server requests Host specifies the host on which the
// URL is sought. Per RFC 2616, this is either the value of
// the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
//
// For client requests Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
Since under the hood ReverseProxy is using this Request to make a client request (after ReverseProxy.Director optionally modifies it), if the Host is set it will override the Host header. This will always be set, because as the first part of the doc comment states "For server requests, Host specifies the host on which the URL is sought".
So in addition to Sebastián's answer, you also need to set req.Host. For example, to proxy to example.com:
proxy := ReverseProxy{
Director: func(req *http.Request) {
req.URL.Scheme = "http"
req.URL.Host = "example.com"
req.Host = "example.com"
}
}
Alternatively you can set req.Host to "" and it will use the value of req.URL.Host.
I figured this out by reading: https://github.com/golang/go/issues/14413

Resources