Adding basic authentication to Go based reverse proxy server - go

I want to secure Docker daemon REST API using Go reverse proxy server. I found this article very relevant. I have never used Go so not sure how to implement basic authentication to this with static username and password. I tried all possible ways i happened to find over Google but none worked for me.
Could some please help adding static basicAuth authentication to following code so that request so that Docker daemon API is only reachable if the request includes username and password:
https://github.com/ben-lab/blog-material/blob/master/golang-reverse-proxy-2/reverse-proxy.go
package main
import (
"fmt"
"io"
"log"
"net/http"
"time"
"github.com/tv42/httpunix"
)
func handleHTTP(w http.ResponseWriter, req *http.Request) {
fmt.Printf("Requested : %s\n", req.URL.Path)
u := &httpunix.Transport{
DialTimeout: 100 * time.Millisecond,
RequestTimeout: 1 * time.Second,
ResponseHeaderTimeout: 1 * time.Second,
}
u.RegisterLocation("docker-socket", "/var/run/docker.sock")
req.URL.Scheme = "http+unix"
req.URL.Host = "docker-socket"
resp, err := u.RoundTrip(req)
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
return
}
defer resp.Body.Close()
copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
}
func copyHeader(dst, src http.Header) {
for k, vv := range src {
for _, v := range vv {
dst.Add(k, v)
}
}
}
func main() {
server := &http.Server{
Addr: ":8888",
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handleHTTP(w, r) }),
}
log.Fatal(server.ListenAndServe())
}
https://github.com/ben-lab/blog-material/blob/master/golang-reverse-proxy-2/reverse-proxy.go

You can access the basic auth header values by calling BasicAuth() on your
req *http.Request object
like:
user, pass, _ := req.BasicAuth()
Then compare user and pass with the static values you have.
https://golang.org/pkg/net/http/#Request.BasicAuth
Update:
func handleHTTP(w http.ResponseWriter, req *http.Request) {
user, pass, _ := req.BasicAuth()
if user != "muuser" || pass != "mysecret" {
// you have to import "errors"
http.Error(w, errors.New("not authoized!!"), http. StatusUnauthorized)
return
}
fmt.Printf("Requested : %s\n", req.URL.Path)
u := &httpunix.Transport{
DialTimeout: 100 * time.Millisecond,
RequestTimeout: 1 * time.Second,
ResponseHeaderTimeout: 1 * time.Second,
}
u.RegisterLocation("docker-socket", "/var/run/docker.sock")
req.URL.Scheme = "http+unix"
req.URL.Host = "docker-socket"
resp, err := u.RoundTrip(req)
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
return
}
defer resp.Body.Close()
copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
}

Here you are, you can copy the logic from my following little project.
https://github.com/alessiosavi/StreamingServer/blob/0f65dbfc77f667777d3047fa1a6b1a2cbd8aaf26/auth/authutils.go
In first instance you need a server for store the users (I've used Redis).
Than you need 3 function for the user
LoginUser
RegisterUser
DeleteUser
During the login/register phase, you generate a cookie hashing username/password and setting the cookie into a Redis table
Than you verify every time that an API is called.
Feel free to copy the code that you need.
Open an issue if something is not well understandable.

Related

Google authentication with http.Redirect

I am trying to implement google authentication using Go.
The code is working fine if I am running a single service which talks to googleoAuth URL.
But I am facing when I am trying to use HTTP.Redirect.
Basically the idea is on receiving a particular URL the first service should redirect the request to some other service which takes care of Google authentication completely.
I am running both services on my same system on two different ports.
Both the server are running as secured server with self signed certificate.
Here are the codes I am using.
main() function for both servers, only the port numbers are different.
func main(){
//Declare the Mux for the server
fmt.Println("Starting service")
mux := handlers.New()
setupHandlers(mux)
certFile := flag.String("certfile", "../cert/certbundle.pem", "certificate PEM file")
keyFile := flag.String("keyfile", "../cert/server.key", "key PEM file")
flag.Parse()
server := http.Server{
Addr: "127.0.0.1:8080",
Handler: mux,
TLSConfig: &tls.Config{
MinVersion: tls.VersionTLS13,
PreferServerCipherSuites: true,
},
}
server.ListenAndServeTLS(*certFile, *keyFile)
}
Here is the redirection code
func New() *http.ServeMux{
mux := http.NewServeMux()
// Root
//mux.Handle("/", http.FileServer(http.Dir("templates/")))
mux.HandleFunc("/", loginPageHandler)
mux.HandleFunc("/internal", internalPageHandler)
// OauthGoogle
mux.HandleFunc("/auth/google/login", oauthGoogleLogin)
mux.HandleFunc("/auth/google/callback", oauthGoogleCallback)
return mux
}
func loginPageHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "https://127.0.0.1:8087/", http.StatusMovedPermanently)
}
Here are the codes for handling google authentication
func loginPageHandler(w http.ResponseWriter, r *http.Request) {
tmpl, err := template.ParseFiles("templates/login.html")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
err = tmpl.Execute(w, nil)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
func oauthGoogleLogin(w http.ResponseWriter, r *http.Request) {
// Create oauthState cookie
oauthState := generateStateOauthCookie(w)
/*
AuthCodeURL receive state that is a token to protect the user from CSRF attacks. You must always provide a non-empty string and
validate that it matches the state query parameter on your redirect callback.
*/
u := googleOauthConfig.AuthCodeURL(oauthState)
http.Redirect(w, r, u, http.StatusTemporaryRedirect)
}
func generateStateOauthCookie(w http.ResponseWriter) string {
var expiration = time.Now().Add( 1 * time.Hour)
b := make([]byte, 16)
rand.Read(b)
state := base64.URLEncoding.EncodeToString(b)
cookie := http.Cookie{Name: "oauthstate", Value: state, Expires: expiration}
http.SetCookie(w, &cookie)
return state
}
func oauthGoogleCallback(w http.ResponseWriter, r *http.Request) {
// Read oauthState from Cookie
fmt.Println("here")
oauthState, _ := r.Cookie("oauthstate")
if r.FormValue("state") != oauthState.Value {
log.Println("invalid oauth google state")
http.Redirect(w, r, "/", http.StatusTemporaryRedirect)
return
}
data, err := getUserDataFromGoogle(r.FormValue("code"))
if err != nil {
log.Println(err.Error())
http.Redirect(w, r, "/", http.StatusTemporaryRedirect)
return
}
fmt.Fprintf(w, "UserInfo: %s\n", data)
}
There are two issues observed.
First I am getting TLS error from the second service which is the destination of the redirection, http: TLS handshake error from 127.0.0.1:57715: remote error: tls: unknown certificate as soon the request is redirected to the second service.
Secondly, in the func oauthGoogleCallback(), the call r.Cookie("oauthstate") fails with no named cookie found error.
Any thought the reason behind TLS error? I believe if that can be solved then rest will work automatically.
Any help is highly appreciated.

Go net/http server error: accept tcp [::]:443: accept4: too many open files; retrying

Here is my server :
package main
import (
"my-project/pkg/configuration"
"my-project/pkg/logger"
"my-project/pkg/server/appConfig"
"my-project/pkg/server/handlers"
"net/http"
"os"
"strings"
)
func main() {
if len(os.Args) < 2 {
logger.Log("error", "main", "Missing config.json file path as argument")
return
}
configuration := configuration.Configuration{}
appConfig.InitConfig(os.Args[1], &configuration)
// download file
http.HandleFunc("/file-download", handlers.DownloadFile(&configuration))
// upload file
http.HandleFunc("/file-upload", handlers.UploadFile(&configuration))
// Get url
http.HandleFunc("/file-url", handlers.GetUrl(&configuration))
// Delete
http.HandleFunc("/delete", handlers.DeleteHandler(&configuration))
// file system
fs := http.FileServer(http.Dir(configuration.RootStoragePath))
corsFS := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/") {
http.NotFound(w, r)
return
}
w.Header().Add("Access-Control-Allow-Origin", "*")
fs.ServeHTTP(w, r)
})
http.Handle("/", corsFS)
err := http.ListenAndServeTLS(":443", "crt/server.crt", "crt/server.key", nil)
if err != nil {
logger.Log("error", "ListenAndServeTLS", err.Error())
}
}
The server is under medium load.
The server crashed after a day of running,
I got the following error:
http: Accept error: accept tcp [::]:443: accept4: too many open files; retrying
The command :
ls -ltr /proc/{PROCESS_ID}/fd
And the list of file, and socket:[XXXXXX] is growing all the time.
I don't want to change ulimit (1024), I don't think it is a long terme fix...
I don't really see where the problem could come from... In the handlers, I manipulate files but I take care to do defer Close()...
Do I have to set timeouts? If so where?
Thank you in advance for all the help ...
I finally managed to find a solution.
The fact is that the http.ListenAndServe method of the net/http package has no timeout by default. This is voluntary from the Go team. So for a service in production, it is necessary to declare a http.Server{} and to configure it. Go doc.
Source : Cloudflare blog post
srv := &http.Server{
Addr: ":443",
ReadTimeout: 30 * time.Second,
WriteTimeout: 120 * time.Second,
}
srv.SetKeepAlivesEnabled(false)
err := srv.ListenAndServeTLS("crt/server.crt", "crt/server.key")
http.DefaultServeMux is the default request multiplexer, and HandleFunc registers the handler function for the given pattern in the DefaultServeMux.
Here is the implementation :
func main() {
if len(os.Args) < 2 {
logger.Log("error", "main", "Missing config.json file path as argument")
return
}
configuration := configuration.Configuration{}
appConfig.InitConfig(os.Args[1], &configuration)
// download file
http.HandleFunc("/file-download", handlers.DownloadFile(&configuration))
// upload file
http.HandleFunc("/file-upload", handlers.UploadFile(&configuration))
// Get url
http.HandleFunc("/file-url", handlers.GetUrl(&configuration))
// Delete
http.HandleFunc("/delete", handlers.DeleteHandler(&configuration))
// file system
fs := http.FileServer(http.Dir(configuration.RootStoragePath))
corsFS := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/") {
http.NotFound(w, r)
return
}
w.Header().Add("Access-Control-Allow-Origin", "*")
fs.ServeHTTP(w, r)
})
http.Handle("/", corsFS)
srv := &http.Server{
Addr: ":443",
ReadTimeout: 30 * time.Second,
WriteTimeout: 120 * time.Second,
}
srv.SetKeepAlivesEnabled(false)
err := srv.ListenAndServeTLS("crt/server.crt", "crt/server.key")
if err != nil {
logger.Log("error", "ListenAndServeTLS", err.Error())
os.Exit(1)
}
}
If you have any other advice, I'm obviously interested.

write health check endpoints if the service has no HTTP server

I want to write health check endpoints for 2 different services, but the problem is they have no HTTP server.
if I can write health check endpoints how can I proceed. or is it mandatory to have an HTTP server to work on health check endpoints with Golang.
Yes, you can add an HTTP health check handler to your application with something like this. Then, in the service that's performing the health check, just make sure it knows which port to run the HTTP checks against.
package main
import "net/http"
func main() {
// Start the health check endpoint and make sure not to block
go func() {
_ = http.ListenAndServe(":8080", http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte("ok"))
},
))
}()
// Start my application code
}
Alternatively, if you need to expose your health check route at a separate path, you can do something like this.
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte("ok"))
})
_ = http.ListenAndServe(":8080", nil)
Updated
If you want to check the health of a go-routine, you can do something like this.
package main
func main() {
crashed := make(chan struct{})
go func() {
defer close(crashed)
}()
select {
case <-crashed:
// Do something now that the go-routine crashed
}
}
It's not mandatory to have an HTTP server.
You can ping the IP address of your service server. For example I use ping repo:
package main
import (
"fmt"
"time"
"github.com/go-ping/ping"
)
func main() {
t := time.NewTicker(5 * time.Second)
for {
select {
case <-t.C:
err := checkService("google", "216.239.38.120")
if err != nil {
fmt.Println("notif to email, error:", err.Error())
time.Sleep(1 * time.Hour) // to not spam email
}
}
}
}
func checkService(name string, ip string) error {
p, err := ping.NewPinger(ip)
if err != nil {
return err
}
p.Count = 3
p.Timeout = 5 * time.Second
err = p.Run()
if err != nil {
return err
}
stats := p.Statistics()
if stats.PacketLoss == 100 {
return fmt.Errorf("service %s down", name)
}
fmt.Printf("stats: %#v\n", stats)
return nil
}

Rate limiter with gorilla mux

I am trying to implement http request limiter to allow 10 request per second per user by their usernames.
At the max 10 request can be hit to the server including requests which are under processing.
Below is what I have implemented with reference of rate-limit.
func init() {
go cleanupVisitors()
}
func getVisitor(username string) *rate.Limiter {
mu.Lock()
defer mu.Unlock()
v, exists := visitors[username]
if !exists {
limiter := rate.NewLimiter(10, 3)
visitors[username] = &visitor{limiter, time.Now()}
return limiter
}
v.lastSeen = time.Now()
return v.limiter
}
func cleanupVisitors() {
for {
time.Sleep(time.Minute)
mu.Lock()
for username, v := range visitors {
if time.Since(v.lastSeen) > 1*time.Minute {
delete(visitors, username)
}
}
mu.Unlock()
}
}
func limit(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mappedArray := hotelapi.SearchResponse{}
mappedArray.StartTime = time.Now().Format("2006-02-01 15:04:05.000000")
mappedArray.EndTime = time.Now().Format("2006-02-01 15:04:05.000000")
userName := r.FormValue("username")
limiter := getVisitor(userName)
if !limiter.Allow() {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusTooManyRequests)
mappedArray.MessageInfo = http.StatusText(http.StatusTooManyRequests)
mappedArray.ErrorCode = strconv.Itoa(http.StatusTooManyRequests)
json.NewEncoder(w).Encode(mappedArray)
return
}
next.ServeHTTP(w, r)
})
}
func route() {
r := mux.NewRouter()
r.PathPrefix("/hello").HandlerFunc(api.ProcessHello).Methods("GET")
ws := r.PathPrefix("/index.php").HandlerFunc(api.ProcessWs).Methods("GET", "POST").Subrouter()
r.Use(panicRecovery)
ws.Use(limit)
http.HandleFunc("/favicon.ico", faviconHandler)
if config.HTTPSEnabled {
err := http.ListenAndServeTLS(":"+config.Port, config.HTTPSCertificateFilePath, config.HTTPSKeyFilePath, handlers.CompressHandlerLevel(r, gzip.BestSpeed))
if err != nil {
fmt.Println(err)
log.Println(err)
}
} else {
err := http.ListenAndServe(":"+config.Port, handlers.CompressHandler(r))
if err != nil {
fmt.Println(err)
log.Println(err)
}
}
}
I have couple of concerns here.
I want limiter only for /index.php and not for /hello. I did implement with Sub route. Is it correct way?
The limit middle ware is not limiting as I assumed. It allows 1 successful request all other requests are returned with too many requests error.
What am I missing here. ?
the subrouter pattern is a solution gorilla proposes , small organizational suggestion though:
r := mux.NewRouter()
r.HandlerFunc("/hello", api.ProcessHello).Methods("GET")
r.HandleFunc("/favicon.ico", faviconHandler)
r.Use(panicRecovery)
ws := r.PathPrefix("/index.php").Subrouter()
ws.Use(limit)
ws.HandlerFunc(api.ProcessWs).Methods("GET", "POST")
you seem to be calling your middleware not only via the Use() method but also calling it over the handler on ListenAndServe, I also see from gorilla same example that a more clear way to approach this is:
server := &http.Server{
Addr: "0.0.0.0:8080",
// Good practice to set timeouts to avoid Slowloris attacks.
WriteTimeout: time.Second * 15,
ReadTimeout: time.Second * 15,
IdleTimeout: time.Second * 60,
Handler: router, // Pass our instance of gorilla/mux in.
}
fmt.Println("starting server")
if err := server.ListenAndServe(); err != nil {
fmt.Println(err)
}
Also, from your source, the pattern of rate limiting you are implementing is to rate limit per user, but you use usernames instead of their IPs to limit their requests, and your question begins without clarifying if you wish to ratelimit per user or rate limit how many requests can be done to the endpoint overall - so maybe you might be getting unexpected behavior due to that too.

how to make a proxy by golang

I'm trying to make a proxy by golang.
The origin version is written by lua, nginx like this:
location / {
keepalive_timeout 3600s;
keepalive_requests 30000;
rewrite_by_lua_file ./test.lua;
proxy_pass http://www.example.com/bd/news/home;
}
and lua file like this:
local req_params = ngx.req.get_uri_args()
local args = {
media = 24,
submedia = 46,
os = req_params.os,
osv = req_params.osv,
make = req_params.make,
model = req_params.model,
devicetype = req_params.devicetype,
conn = req_params.conn,
carrier = req_params.carrier,
sw = req_params.w,
sh = req_params.h,
}
if tonumber(req_params.os) == 1 then
args.imei = req_params.imei
args.adid = req_params.android_id
end
ngx.req.set_uri_args(args)
I try to do the same thing by golang, and my code is like this:
const newsTargetURL = "http://www.example.com/bd/news/home"
func GetNews(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only get allowed", http.StatusMethodNotAllowed)
return
}
// deal params
rq := r.URL.Query()
os := rq.Get("os")
osv := rq.Get("osv")
imei := rq.Get("imei")
androidID := rq.Get("android_id")
deviceMake := rq.Get("make")
model := rq.Get("model")
deviceType := rq.Get("devicetype")
sw := rq.Get("w")
sh := rq.Get("h")
conn := rq.Get("conn")
carrier := rq.Get("carrier")
uv := make(url.Values)
uv.Set("media", "24")
uv.Set("submedia", "46")
uv.Set("os", os)
uv.Set("osv", osv)
if os == "1" {
uv.Set("imei", imei)
uv.Set("anid", androidID)
}
uv.Set("make", deviceMake)
uv.Set("model", model)
uv.Set("sw", sw)
uv.Set("sh", sh)
uv.Set("devicetype", deviceType)
uv.Set("ip", ip)
uv.Set("ua", ua)
uv.Set("conn", conn)
uv.Set("carrier", carrier)
t := newsTargetURL + "?" + uv.Encode()
// make a director
director := func(req *http.Request) {
u, err := url.Parse(t)
if err != nil {
panic(err)
}
req.URL = u
}
// make a proxy
proxy := &httputil.ReverseProxy{Director: director}
proxy.ServeHTTP(w, r)
}
func main() {
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(GetNews))
srv := &http.Server{
Addr: ":2222",
Handler: mux,
}
srv.ListenAndServe()
}
I put this go version to the same server where lua version locate, but it does not work as lua file do. I read the httputil document but found nothing that can help. What do I need to do?
I wrote together a simple proxy for GET requests. Hope this helps.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
const newsTargetURL = "http://www.example.com/bd/news/home"
func main() {
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(GetNews))
srv := &http.Server{
Addr: ":2222",
Handler: mux,
}
// output error and quit if ListenAndServe fails
log.Fatal(srv.ListenAndServe())
}
func GetNews(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only get allowed", http.StatusMethodNotAllowed)
return
}
// build proxy url
urlstr := fmt.Sprintf("%s?%s", newsTargetURL, r.URL.RawQuery)
// request the proxy url
resp, err := http.Get(urlstr)
if err != nil {
http.Error(w, fmt.Sprintf("error creating request to %s", urlstr), http.StatusInternalServerError)
return
}
// make sure body gets closed when this function exits
defer resp.Body.Close()
// read entire response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
http.Error(w, "error reading response body", http.StatusInternalServerError)
return
}
// write status code and body from proxy request into the answer
w.WriteHeader(resp.StatusCode)
w.Write(body)
}
You can try it as is. It will work and show the content of example.com.
It uses a single handler GetNews for all requests. It skips all of the request parameter parsing and building by simply using r.url.RawQuery and newsTargetURL to build the new url.
Then we make a request to the new url (the main part missing in your question). From the response we read resp.StatusCode and resp.body to use in our response to the original request.
The rest is error handling.
The sample does not forward any additional information like cookies, headers, etc. That can be added as needed.

Resources