I'm using Envoy as mTLS proxies on load-balanced servers.
The Go client makes an HTTPS call to the load balancer which responds with a redirect to another server. I need to modify the port number in the response to use the Envoy port.
Is it possible to read Response.Header.Get("Location") and create a new response in order to change the port?
The docs say
// RoundTrip should not attempt to interpret the response.
It would look like
type EnvoyRoundTripper struct {
PortMapper http.RoundTripper
}
func (ert EnvoyRoundTripper) RoundTrip(req *http.Request) (res *http.Response, e error) {
res, e = ert.PortMapper.RoundTrip(req)
if e == nil && res.StatusCode == http.StatusTemporaryRedirect {
redirect := res.Header.Get("Location")
// res = Create a new Response with a modified redirect
}
return
}
Related
I'm trying to write some simple load balancer on Go. I'm using simple http.Server and httputil.NewSingleHostReverseProxy like so:
func main() {
...
server := http.Server{
Addr: ":8080",
ConnState: connStateHook,
Handler: http.HandlerFunc(loadBalance),
}
}
HandlerFunc is as follows:
func loadBalance(w http.ResponseWriter, r *http.Request) {
// let's imagine I have a hardcoded proxy to simplify things
serverURL, _ := url.Parse("http://127.0.0.1:5000")
proxy := httputil.NewSingleHostReverseProxy(serverURL)
proxy.ServeHTTP(w, r)
}
So whenever request will be sent to my localhost:8080 it'll be proxied to 127.0.0.1:5000.
My goal is to handle how many active request's are handled by that proxy endpoint. I know there is http.Server.ConnState hook which allows you to track state of http request and fire "callbacks".
I've made a simple function to store the number of connections in global variable:
var connectionsAmount int
func connStateHook(connection net.Conn, state http.ConnState) {
if state == http.StateActive {
connectionsAmount = connectionsAmount + 1
} else if state == http.StateIdle {
connectionsAmount = connectionsAmount - 1
} else if state == http.StateClosed {
connectionsAmount = connectionsAmount - 1
}
}
It works pretty well until I add additional N endpoints, because now I can't pair a request with specific proxy server.
And I want to track this number, let's say, in Endpoint struct:
type Endpoint struct {
URL *url.URL
Proxy *httputil.ReverseProxy
ActiveConnections int
}
In what way I can implement such tracking?
I have reverse proxies in my main web-server that are dedicated to a certain micro-service and handle forward requests to their appropriate micro-services.
func newTrimPrefixReverseProxy(target *url.URL, prefix string) *httputil.ReverseProxy {
director := func(req *http.Request) {
// ... trims prefix from request path and prepends the path of the target url
}
return &httputil.ReverseProxy{Director: director}
}
This has worked perfectly for pure JSON responses, but I have ran into issues recently when trying to serve content (stream responses) through the reverse proxy. The means for serving the content is irrelevant, the (video) content is served as intended when the service is accessed directly and not through the reverse proxy.
Serving the content:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
http.ServeContent(w, r, "video.mp4", time.Now().Add(time.Hour*24*365*12*9*-1), videoReadSeeker)
})
Again, the videoReadSeeker and how the content is served is not the issue, the issue is having my response relayed as intended to the requester through the reverse proxy; when accessing the service directly, the video shows up and I can scrub it to my heart's content.
Note that the response for data the content is received (http status, headers), but the content stream in the response body is not.
How can I make sure that the reverse proxy handles streamed responses as intended for the content?
Do you get the same results when using:
package main
import (
"log"
"net/http"
"net/http/httputil"
"net/url"
)
func main() {
u, err := url.Parse("http://localhost:8080/asdfasdf")
if err != nil {
log.Fatal("url.Parse: %v", err)
}
proxy := httputil.NewSingleHostReverseProxy(u)
log.Printf("Listening at :8081")
if err := http.ListenAndServe(":8081", proxy); err != nil {
log.Fatal("ListenAndServe: %v", err)
}
}
Ultimately these are the same implementation under the hood, but the director provided here ensures that some of the expected headers exist that you will need for some proxy features to function as expected.
The go HTTP client is adding a "chunked" transfer encoding field to my client request. Unfortunately, this is not supported by the service I'm connecting to and it comes back with an error.
Is there a way to disable this?
This is my Request code:
// DoHTTPRequest Do a full HTTP Client Request, with timeout
func DoHTTPRequest(method, url string, body io.Reader, headers map[string]string, timeout time.Duration) (*http.Response, error) {
// Create the request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// Add headers
for k, v := range headers {
req.Header.Set(k, v)
}
client := &http.Client{
Timeout: timeout,
}
return client.Do(req)
}
Basically I would like this header dropped. This client is talking to S3 which is rather sensitive as to what headers it is sent.
I get this error:
A header you provided implies functionality that is not implemented
TransferEncoding is a field directly on the Request struct. If you set it explicitly it will not be overridden.
req.TransferEncoding = []string{"identity"}
https://golang.org/pkg/net/http/#Request
Setting Transfer Encoding header like this makes it not use chunked:
req.TransferEncoding = []string{"identity"}
However the http client sources give the reason for it choosing chunked in my case. Specifically, I was using "PUT" as the method and had no content-length specified. So all I needed was to set the req.ContentLength.
However, you can see my DoHTTPRequest wrapper function doesn't know how big to set it. And I had assumed that setting the header will make it work originally. Well, it doesn't work by setting the header. And you can see why in the sources that determine whether to make it use chunked encoding.
// shouldSendChunkedRequestBody reports whether we should try to send a
// chunked request body to the server. In particular, the case we really
// want to prevent is sending a GET or other typically-bodyless request to a
// server with a chunked body when the body has zero bytes, since GETs with
// bodies (while acceptable according to specs), even zero-byte chunked
// bodies, are approximately never seen in the wild and confuse most
// servers. See Issue 18257, as one example.
//
// The only reason we'd send such a request is if the user set the Body to a
// non-nil value (say, ioutil.NopCloser(bytes.NewReader(nil))) and didn't
// set ContentLength, or NewRequest set it to -1 (unknown), so then we assume
// there's bytes to send.
//
// This code tries to read a byte from the Request.Body in such cases to see
// whether the body actually has content (super rare) or is actually just
// a non-nil content-less ReadCloser (the more common case). In that more
// common case, we act as if their Body were nil instead, and don't send
// a body.
func (t *transferWriter) shouldSendChunkedRequestBody() bool {
// Note that t.ContentLength is the corrected content length
// from rr.outgoingLength, so 0 actually means zero, not unknown.
if t.ContentLength >= 0 || t.Body == nil { // redundant checks; caller did them
return false
}
if requestMethodUsuallyLacksBody(t.Method) {
// Only probe the Request.Body for GET/HEAD/DELETE/etc
// requests, because it's only those types of requests
// that confuse servers.
t.probeRequestBody() // adjusts t.Body, t.ContentLength
return t.Body != nil
}
// For all other request types (PUT, POST, PATCH, or anything
// made-up we've never heard of), assume it's normal and the server
// can deal with a chunked request body. Maybe we'll adjust this
// later.
return true
}
So my solution is simply:
// DoHTTPRequest Do a full HTTP Client Request, with timeout
func DoHTTPRequest(method, url string, body io.Reader, headers map[string]string, timeout time.Duration) (*http.Response, error) {
// Create the request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// Add headers
for k, v := range headers {
req.Header.Set(k, v)
// Set the Content Length correctly if specified.
if strings.EqualFold(k, "Content-Length") {
length, err := strconv.Atoi(v)
if err != nil {
return nil, fmt.Errorf("Bad Content-Length header")
}
req.ContentLength = int64(length)
}
}
client := &http.Client{
Timeout: timeout,
Transport: &loghttp.Transport{},
}
return client.Do(req)
}
Which satisfies S3 as far as having a correct content length. I didn't need to set the TransferEncoding to identity.
Brand new to Go.. Still obviously learning the syntax and the basics.. But I do have a specific goal in mind..
I'm trying to just get a simple server up on :8080 that can respond to both HTTP and socket.io (via /socket.io/ url), specificaly with CORS.
My code:
package main
import (
"log"
"net/http"
"github.com/rs/cors"
"github.com/googollee/go-socket.io"
)
func SayHelloWorld(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello, World!"))
}
func main() {
c := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
AllowCredentials: true,
})
server, err := socketio.NewServer(nil)
if err != nil {
log.Fatal(err)
}
server.On("connection", func(so socketio.Socket) {
log.Println("on connection")
so.Join("chat")
so.On("chat message", func(msg string) {
log.Println("emit:", so.Emit("chat message", msg))
so.BroadcastTo("chat", "chat message", msg)
})
so.On("disconnection", func() {
log.Println("on disconnect")
})
})
server.On("error", func(so socketio.Socket, err error) {
log.Println("error:", err)
})
http.Handle("/socket.io/", c.Handler(server))
http.HandleFunc("/", SayHelloWorld)
log.Println("Serving at localhost:8080...")
log.Fatal(http.ListenAndServe(":8080", nil))
}
On the client side I'm still seeing:
WebSocket connection to 'wss://api.domain.com/socket.io/?EIO=3&transport=websocket&sid=xNWd9aZvwDnZOrXkOBaC' failed: WebSocket is closed before the connection is established.
(index):1 XMLHttpRequest cannot load https://api.domain.com/socket.io/?EIO=3&transport=polling&t=1420662449235-3932&sid=xNWd9aZvwDnZOrXkOBaC. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://fiddle.jshell.net' is therefore not allowed access.
EDIT #1:
So I've been banging my head away trying to understand why I can't connect.. Came across an even more confusing piece of the puzzle?
https://gist.github.com/acoyfellow/167b055da85248c94fc4
The above gist is the code of my golang server + the browser code used to connect.. This code will send 30 HTTP GET requests per second to the backend, without connecting, upgrading, or giving any errors (client or server side).. it essentially DDOS's my own backend?
Someone, please someone tell me I'm doing something stupid.. This is quite the pickle :P
EDIT #2:
I can stop the "DDOS" by simply adjusting the trailing / on the URL of the socket.io endpoint in Go.. So: mux.Handle("/socket.io", server) to mux.Handle("/socket.io/", server) will now produce error messages and connection attempts with error responses of:
WebSocket connection to 'wss://api.domain.com/socket.io/?EIO=3&transport=websocket&sid=0TzmTM_QtF1TaS4exiwF' failed: Error during WebSocket handshake: Unexpected response code: 400 socket.io-1.2.1.js:2
GET https://api.domain.com/socket.io/?EIO=3&transport=polling&t=1420743204485-62&sid=0TzmTM_QtF1TaS4exiwF 400 (Bad Request)
So I gave up using googoolee's Socket.io implementation and went with gorilla's.
I checked out their examples: https://github.com/gorilla/websocket/tree/master/examples/chat
Checked out their docs: http://www.gorillatoolkit.org/pkg/websocket
-- Under Origin Considerations I found:
An application can allow connections from any origin by specifying a function that always returns true:
var upgrader = websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool { return true },
}
I added this CheckOrigin function to the conn.go file in their example, and was able to get a CORS socket server talking to a browser.
As a first adventure into Golang, this was frustrating and fun.. +1 to anyone else learning
Don't you mean http + ws or https + wss. If you remove a s from wss, you should be able to connect.
If you want tls for web socket (wss), then you need to http.ListenAndServeTLS.
It appears that CORS does not apply to WebSockets. Per this related question "With WebSocket, there is an "origin" header, which browser MUST fill with the origin of the HTML containing the JS that opens the WS connection."
As stated here:
Cross origin websockets with Golang
How about in your SayHelloWorld func, adding something like:
w.Header().Set("Access-Control-Allow-Origin", "*")
Or, possibly better:
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
}
I get the similar problerm with normal ajax call. It require more work in both front-end and backend. I belive most popular front-end libs liek JQuery or AngularJS handle these very well.
I see you're using the https://github.com/rs/cors package but you don't include the usage of that package, here is the implement with only Go std package:
type CrossOriginServer struct {}
func (s *CrossOriginServer) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
// you may need to add some more headers here
allowHeaders := "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization"
if origin := req.Header.Get("Origin"); validOrigin(origin) {
rw.Header().Set("Access-Control-Allow-Origin", origin)
rw.Header().Set("Access-Control-Allow-Methods", "POST, PUT, PATCH, GET, DELETE")
rw.Header().Set("Access-Control-Allow-Headers", allowHeaders)
}
if req.Method == "OPTIONS" {
return
}
// if you want, you can use gorilla/mux or any routing package here
mux := http.NewServeMux()
mux.Handle("/socket.io/", c.Handler(server))
mux.HandleFunc("/", SayHelloWorld)
mux.ServeHTTP(rw, req)
}
func validOrigin(origin string) bool {
allowOrigin := []string{
"http://localhost:8081",
"http://example.com"
}
for _, v := range allowOrigin {
if origin == v {
return true
}
}
return false
}
func main() {
// do you stuff
// ...
// ...
http.ListenAndServe(":8080", &CrossOriginServer{})
}
I built an HTTP server. I am using the code below to get the request URL, but it does not get full URL.
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Printf("Req: %s %s", r.URL.Host, r.URL.Path)
}
I only get "Req: / " and "Req: /favicon.ico".
I want to get full client request URL as "1.2.3.4/" or "1.2.3.4/favicon.ico".
Thanks.
From the documentation of net/http package:
type Request struct {
...
// The host on which the URL is sought.
// Per RFC 2616, this is either the value of the Host: header
// or the host name given in the URL itself.
// It may be of the form "host:port".
Host string
...
}
Modified version of your code:
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Printf("Req: %s %s\n", r.Host, r.URL.Path)
}
Example output:
Req: localhost:8888 /
I use req.URL.RequestURI() to get the full url.
From net/http/requests.go :
// RequestURI is the unmodified Request-URI of the
// Request-Line (RFC 2616, Section 5.1) as sent by the client
// to a server. Usually the URL field should be used instead.
// It is an error to set this field in an HTTP client request.
RequestURI string
If you detect that you are dealing with a relative URL (r.URL.IsAbs() == false), you sill have access to r.Host (see http.Request), the Host itself.
Concatenating the two would give you the full URL.
Generally, you see the reverse (extracting Host from an URL), as in gorilla/reverse/matchers.go
// getHost tries its best to return the request host.
func getHost(r *http.Request) string {
if r.URL.IsAbs() {
host := r.Host
// Slice off any port information.
if i := strings.Index(host, ":"); i != -1 {
host = host[:i]
}
return host
}
return r.URL.Host
}