I built an HTTP server. I am using the code below to get the request URL, but it does not get full URL.
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Printf("Req: %s %s", r.URL.Host, r.URL.Path)
}
I only get "Req: / " and "Req: /favicon.ico".
I want to get full client request URL as "1.2.3.4/" or "1.2.3.4/favicon.ico".
Thanks.
From the documentation of net/http package:
type Request struct {
...
// The host on which the URL is sought.
// Per RFC 2616, this is either the value of the Host: header
// or the host name given in the URL itself.
// It may be of the form "host:port".
Host string
...
}
Modified version of your code:
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Printf("Req: %s %s\n", r.Host, r.URL.Path)
}
Example output:
Req: localhost:8888 /
I use req.URL.RequestURI() to get the full url.
From net/http/requests.go :
// RequestURI is the unmodified Request-URI of the
// Request-Line (RFC 2616, Section 5.1) as sent by the client
// to a server. Usually the URL field should be used instead.
// It is an error to set this field in an HTTP client request.
RequestURI string
If you detect that you are dealing with a relative URL (r.URL.IsAbs() == false), you sill have access to r.Host (see http.Request), the Host itself.
Concatenating the two would give you the full URL.
Generally, you see the reverse (extracting Host from an URL), as in gorilla/reverse/matchers.go
// getHost tries its best to return the request host.
func getHost(r *http.Request) string {
if r.URL.IsAbs() {
host := r.Host
// Slice off any port information.
if i := strings.Index(host, ":"); i != -1 {
host = host[:i]
}
return host
}
return r.URL.Host
}
Related
I'm using Envoy as mTLS proxies on load-balanced servers.
The Go client makes an HTTPS call to the load balancer which responds with a redirect to another server. I need to modify the port number in the response to use the Envoy port.
Is it possible to read Response.Header.Get("Location") and create a new response in order to change the port?
The docs say
// RoundTrip should not attempt to interpret the response.
It would look like
type EnvoyRoundTripper struct {
PortMapper http.RoundTripper
}
func (ert EnvoyRoundTripper) RoundTrip(req *http.Request) (res *http.Response, e error) {
res, e = ert.PortMapper.RoundTrip(req)
if e == nil && res.StatusCode == http.StatusTemporaryRedirect {
redirect := res.Header.Get("Location")
// res = Create a new Response with a modified redirect
}
return
}
In my Go API, I'm using gin, and I have one value set in my Access-Control-Allow-Origin header. If I have more than one, my react UI throws an error to the effect of The Access-Control-Allow-Origin header contains multiple values 'http://value1, http://value2', but only one is allowed.... I need to set multiple values. How do I do this?
The API is a reverse proxy, and here's the relevant code:
func proxy(c *gin.Context) {
var remote = "myUrl"
proxy := httputil.NewSingleHostReverseProxy(remote)
proxy.Director = func(req *http.Request) {
req.Header.Set("Authorization", "My Auth Values")
req.Host = remote.Host
req.URL.Scheme = remote.Scheme
req.URL.Host = remote.Host
}
proxy.ModifyResponse = addCustomHeader
proxy.ServeHTTP(c.Writer, c.Request)
}
func addCustomHeader(r *http.Response) error {
r.Header["Access-Control-Allow-Origin"] = []string{"value1"}
return nil
}
A CORS header can only contain a single value. If you want to implement your own CORS middleware, you need to work around that fact.
A simple CORS middleware will add the Access-Control-Allow-Origin header with the value of the specific address of the incoming request, usually taken from the Referer or Origin header. Typically, you match this against a list or map first, to see if it's in your allow list. If so, then the address of the request is added as allowed origin (as a single value).
A simple example could look like this
allowList := map[string]bool{
"https://www.google.com": true,
"https://www.yahoo.com": true,
}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if origin := r.Header.Get("Origin"); allowList[origin] {
w.Header().Set("Access-Control-Allow-Origin", origin)
}
})
Since you are using the reverse proxy, you can access the request from the response.
mod := func(allowList map[string]bool) func(r *http.Response) error {
return func(r *http.Response) error {
if origin := r.Request.Header.Get("Origin"); allowList[origin] {
r.Header.Set("Access-Control-Allow-Origin", origin)
}
return nil
}
}
proxy := &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme = "https"
r.URL.Host = "go.dev"
r.Host = r.URL.Host
},
ModifyResponse: mod(allowList),
}
You only need a single value for each incoming request. The usual technique is to configure trusted origins on the server, eg:
trustedOrigins: [https://www.domain1.com, https://www.domain2.com]
Then check the runtime value of the origin header, which is sent by all modern browsers. If this is a trusted origin then add CORS headers:
Access-Control-Allow-Origin: https://www.domain2.com
A wildcard could be used but that is not recommended and also will not work as intended if you are also using credentialed requests (eg those with cookies).
How can I get the client IP address in Go gRPC from a load balancer that adds x-real-ip and x-forward-for to the header?
The "peer" lib is giving me the only load balancer ip. (from this answer: Get client's IP address from load balance server by X-Forwarded-For)
Just to make it clear: The context where I need to grab this information is inside the GRPC server handler. It may be in an interceptor or inside the request handler:
example:
func (s *myservice) MethodName(ctx context.Context, request *pb.MethodRequest) (*pb.MethodResponse, error) {
// How to get ip from ctx ?
// I have tryied peer but it gives me the loadbalancer ip
peer, ok := peer.FromContext(ctx)
return &pb.MethodResponse{Pong: "ok"}, nil
}
It would be ok if I could get this from the interceptor as well... either way works for me.
func (a *MyInterceptor) Unary() grpc.UnaryServerInterceptor {
return func(
ctx context.Context,
req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
log.Debug("--> My interceptor: " + info.FullMethod)
// How to get ip from ctx ?
return handler(ctx, req)
}
}
PS: I'm using Kong as load balancer. https://docs.konghq.com/0.14.x/loadbalancing/
Looking at the loadbalancer docs you can see that there is a header set of X-Real-IP https://docs.konghq.com/0.14.x/configuration/#real_ip_header
This should be returned in the metadata of the ctx. I can't test this unfortunately, but let me know if it works
func (a *MyInterceptor) Unary() grpc.UnaryServerInterceptor {
return func(
ctx context.Context,
req interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
log.Debug("--> My interceptor: " + info.FullMethod)
// How to get ip from ctx ?
var realIP string
m, ok := metadata.FromIncomingContext(ctx)
if ok {
realIP := md.Get("X-Real-IP")
}
// do what you need with realIP
return handler(ctx, req)
}
}
You can read the headers from the request object.
func handler(w http.ResponseWriter, r *http.Request) {
realIP := r.Header.Get("x-real-ip")
forwardFor := r.Header.Get("x-forward-for");
// ...
}
Brand new to Go.. Still obviously learning the syntax and the basics.. But I do have a specific goal in mind..
I'm trying to just get a simple server up on :8080 that can respond to both HTTP and socket.io (via /socket.io/ url), specificaly with CORS.
My code:
package main
import (
"log"
"net/http"
"github.com/rs/cors"
"github.com/googollee/go-socket.io"
)
func SayHelloWorld(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello, World!"))
}
func main() {
c := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
AllowCredentials: true,
})
server, err := socketio.NewServer(nil)
if err != nil {
log.Fatal(err)
}
server.On("connection", func(so socketio.Socket) {
log.Println("on connection")
so.Join("chat")
so.On("chat message", func(msg string) {
log.Println("emit:", so.Emit("chat message", msg))
so.BroadcastTo("chat", "chat message", msg)
})
so.On("disconnection", func() {
log.Println("on disconnect")
})
})
server.On("error", func(so socketio.Socket, err error) {
log.Println("error:", err)
})
http.Handle("/socket.io/", c.Handler(server))
http.HandleFunc("/", SayHelloWorld)
log.Println("Serving at localhost:8080...")
log.Fatal(http.ListenAndServe(":8080", nil))
}
On the client side I'm still seeing:
WebSocket connection to 'wss://api.domain.com/socket.io/?EIO=3&transport=websocket&sid=xNWd9aZvwDnZOrXkOBaC' failed: WebSocket is closed before the connection is established.
(index):1 XMLHttpRequest cannot load https://api.domain.com/socket.io/?EIO=3&transport=polling&t=1420662449235-3932&sid=xNWd9aZvwDnZOrXkOBaC. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://fiddle.jshell.net' is therefore not allowed access.
EDIT #1:
So I've been banging my head away trying to understand why I can't connect.. Came across an even more confusing piece of the puzzle?
https://gist.github.com/acoyfellow/167b055da85248c94fc4
The above gist is the code of my golang server + the browser code used to connect.. This code will send 30 HTTP GET requests per second to the backend, without connecting, upgrading, or giving any errors (client or server side).. it essentially DDOS's my own backend?
Someone, please someone tell me I'm doing something stupid.. This is quite the pickle :P
EDIT #2:
I can stop the "DDOS" by simply adjusting the trailing / on the URL of the socket.io endpoint in Go.. So: mux.Handle("/socket.io", server) to mux.Handle("/socket.io/", server) will now produce error messages and connection attempts with error responses of:
WebSocket connection to 'wss://api.domain.com/socket.io/?EIO=3&transport=websocket&sid=0TzmTM_QtF1TaS4exiwF' failed: Error during WebSocket handshake: Unexpected response code: 400 socket.io-1.2.1.js:2
GET https://api.domain.com/socket.io/?EIO=3&transport=polling&t=1420743204485-62&sid=0TzmTM_QtF1TaS4exiwF 400 (Bad Request)
So I gave up using googoolee's Socket.io implementation and went with gorilla's.
I checked out their examples: https://github.com/gorilla/websocket/tree/master/examples/chat
Checked out their docs: http://www.gorillatoolkit.org/pkg/websocket
-- Under Origin Considerations I found:
An application can allow connections from any origin by specifying a function that always returns true:
var upgrader = websocket.Upgrader{
CheckOrigin: func(r *http.Request) bool { return true },
}
I added this CheckOrigin function to the conn.go file in their example, and was able to get a CORS socket server talking to a browser.
As a first adventure into Golang, this was frustrating and fun.. +1 to anyone else learning
Don't you mean http + ws or https + wss. If you remove a s from wss, you should be able to connect.
If you want tls for web socket (wss), then you need to http.ListenAndServeTLS.
It appears that CORS does not apply to WebSockets. Per this related question "With WebSocket, there is an "origin" header, which browser MUST fill with the origin of the HTML containing the JS that opens the WS connection."
As stated here:
Cross origin websockets with Golang
How about in your SayHelloWorld func, adding something like:
w.Header().Set("Access-Control-Allow-Origin", "*")
Or, possibly better:
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
}
I get the similar problerm with normal ajax call. It require more work in both front-end and backend. I belive most popular front-end libs liek JQuery or AngularJS handle these very well.
I see you're using the https://github.com/rs/cors package but you don't include the usage of that package, here is the implement with only Go std package:
type CrossOriginServer struct {}
func (s *CrossOriginServer) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
// you may need to add some more headers here
allowHeaders := "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization"
if origin := req.Header.Get("Origin"); validOrigin(origin) {
rw.Header().Set("Access-Control-Allow-Origin", origin)
rw.Header().Set("Access-Control-Allow-Methods", "POST, PUT, PATCH, GET, DELETE")
rw.Header().Set("Access-Control-Allow-Headers", allowHeaders)
}
if req.Method == "OPTIONS" {
return
}
// if you want, you can use gorilla/mux or any routing package here
mux := http.NewServeMux()
mux.Handle("/socket.io/", c.Handler(server))
mux.HandleFunc("/", SayHelloWorld)
mux.ServeHTTP(rw, req)
}
func validOrigin(origin string) bool {
allowOrigin := []string{
"http://localhost:8081",
"http://example.com"
}
for _, v := range allowOrigin {
if origin == v {
return true
}
}
return false
}
func main() {
// do you stuff
// ...
// ...
http.ListenAndServe(":8080", &CrossOriginServer{})
}
I don't know why this reverse proxy is not working. I've seen several examples and I can't find anything wrong with it.
package main
import (
"log"
"net/url"
"net/http"
"net/http/httputil"
)
func report(r *http.Request){
log.Print("URL: " + r.URL.Path)
log.Print("Scheme: " + r.URL.Scheme)
log.Print("Host: " + r.URL.Host)
//r.URL.Scheme = "http"
//r.URL.Host = "stackoverflow.com"
//r.Header.Set("Host", "stackoverflow.com")
//log.Print("Header Host: " + r.Header.Get("Host"))
}
func main() {
proxy := httputil.NewSingleHostReverseProxy( &url.URL{Scheme:"http",Host:"myrealserver.com"})
proxy.Director = report
// http.Handle("/", proxy)
error := http.ListenAndServe("mylocalhost.com:8080", proxy)
if(error != nil) {
log.Fatal(error)
}
}
It logs:
2014/04/18 21:32:50 URL: /arg/es
2014/04/18 21:32:50 Scheme:
2014/04/18 21:32:50 Host:
2014/04/18 21:32:50 http: proxy error: unsupported protocol scheme ""
2014/04/18 21:32:51 URL: /favicon.ico
2014/04/18 21:32:51 Scheme:
2014/04/18 21:32:51 Host:
2014/04/18 21:32:51 http: proxy error: unsupported protocol scheme ""
If I uncomment the line that redefines the Schema the error message becomes:
2014/04/18 21:38:05 http: proxy error: http: no Host in request URL
If I uncomment the line that redefines the host also, then the target server becomes stackoverflow.com (I mean, it never uses "myrealserver.com").
If I ask for mylocalhost.com:8080/somepath (or even /) then I get a 404 from Stackoverflow, no matter if stackoverflow.com/somepath exists or not. It says:
Couldn't find mylocalhost.com:8080
The Q&A site mylocalhost.com:8080 doesn't seem to exist... yet
It does not translate the Host header automatically.
If then I uncomment the line that sets (and the other one that prints) the Header "Host". Then I can read "stackoverflow.com" in the log, but I still get the same 404 page reporting that I am trying to access "mylocalhost.com".
I'm using go1.2.1 linux/amd64
How is it that I am supposed to make the program work as a proxy?
Thanks to Alex from Golang-nuts, I have the answer now.
This is what Alex said:
Just need to set http.Request.Host [and scheme] in the Director to get this
working: http://play.golang.org/p/I17ZSM6LQb
If you read the source for SingleHostReverseProxy
(http://golang.org/src/pkg/net/http/httputil/reverseproxy.go#L61), it
sets its own Director which you are overriding. So you need to
reimplement what it already does plus the extra Host change.
Anyway, that didn't solve de Header part of the problem: the target server was still receiving "localhost:8080" as the HTTP Host name, so I did it without the ReverseProxy package, just with http and a RoundTripper, plus a helper function that copies all the headers:
package main
import (
"flag"
"fmt"
"os"
"log"
"net/http"
"io/ioutil"
)
var target *string
func main() {
target = flag.String("target", "http://stackoverflow.com", "target URL for reverse proxy")
flag.Parse()
http.HandleFunc("/", report)
log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}
func report(w http.ResponseWriter, r *http.Request){
uri := *target+r.RequestURI
fmt.Println(r.Method + ": " + uri)
if r.Method == "POST" {
body, err := ioutil.ReadAll(r.Body)
fatal(err)
fmt.Printf("Body: %v\n", string(body));
}
rr, err := http.NewRequest(r.Method, uri, r.Body)
fatal(err)
copyHeader(r.Header, &rr.Header)
// Create a client and query the target
var transport http.Transport
resp, err := transport.RoundTrip(rr)
fatal(err)
fmt.Printf("Resp-Headers: %v\n", resp.Header);
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fatal(err)
dH := w.Header()
copyHeader(resp.Header, &dH)
dH.Add("Requested-Host", rr.Host)
w.Write(body)
}
func fatal(err error) {
if err != nil {
log.Fatal(err)
os.Exit(1)
}
}
func copyHeader(source http.Header, dest *http.Header){
for n, v := range source {
for _, vv := range v {
dest.Add(n, vv)
}
}
}
Now I'm able to see StackOverflow or any other site how it's supposed to be.
I'm still working on POST calls, though, so this is a work in progress.
A little late to the party, but ReverseProxy isn't broken, it's just a little confusing because it doesn't work how you'd expect (at the least, I expected it to work the way you did, so that makes two of us).
From the docs:
// For server requests Host specifies the host on which the
// URL is sought. Per RFC 2616, this is either the value of
// the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
//
// For client requests Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
Since under the hood ReverseProxy is using this Request to make a client request (after ReverseProxy.Director optionally modifies it), if the Host is set it will override the Host header. This will always be set, because as the first part of the doc comment states "For server requests, Host specifies the host on which the URL is sought".
So in addition to Sebastián's answer, you also need to set req.Host. For example, to proxy to example.com:
proxy := ReverseProxy{
Director: func(req *http.Request) {
req.URL.Scheme = "http"
req.URL.Host = "example.com"
req.Host = "example.com"
}
}
Alternatively you can set req.Host to "" and it will use the value of req.URL.Host.
I figured this out by reading: https://github.com/golang/go/issues/14413