Program doesn't see Host Header - go

Trying to proxy a request, and I discovered the host header is not being present in the headers received
UPDATE: as pointed out from one of the answers, I might need the Host field, this doesn't change either, host headers are still not present.
code:
request, err := http.ReadRequest(bufio.NewReader(conn))
if err != nil {
log.Println("Error reading request", err)
return
}
target, err := net.Dial("tcp", request.URL.Host)
if err != nil {
log.Println(host)
log.Println("Error reaching target", err)
}
defer target.Close()
curl command
curl -v localhost:8080
Output:
Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.79.1
> Accept: */*
>
So host header is present in the curl.
Added a debug step to the program to print headers
fmt.Println("Headers:")
for key, values := range request.Header {
fmt.Printf("%s: %v\n", key, values)
}
Output from program:
Headers:
Accept: [*/*]
User-Agent: [curl/7.79.1]
Still no presence of Host header
Now I tried passing a host header with curl:
curl -v -H 'Host: localhost:3000' localhost:8080
Same output, header present in curl but not on the program

What you want is the Host field of the Request type. From the Godoc:
// For server requests, Host specifies the host on which the
// URL is sought. For HTTP/1 (per RFC 7230, section 5.4), this
// is either the value of the "Host" header or the host name
// given in the URL itself. For HTTP/2, it is the value of the
// ":authority" pseudo-header field.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
// To prevent DNS rebinding attacks, server Handlers should
// validate that the Host header has a value for which the
// Handler considers itself authoritative. The included
// ServeMux supports patterns registered to particular host
// names and thus protects its registered Handlers.
//
// For client requests, Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string

Related

How to change which IP address a go-fiber client is using?

I'm using Fiber as an HTTP client to make some requests to an http server, however I'm being rate limited. On my vm I configured 5 different IP addresses (public/private) and have confirmed that they are indeed connected to the internet.
curl --interface 10.0.0.4 ipinfo.io/json
curl --interface 10.0.0.5 ipinfo.io/json
...
curl --interface 10.0.0.8 ipinfo.io/json
each one returns a different public facing ip address.
Now I'm interested in making round-robin requests using these local addresses but I'm not so sure how to go about it.
Is there some sort of property or function I can set/call to change where the outgoing request is coming from?
I've looked around at fasthttp.HostClient which fiber.Agent extends but I didn't see anything useful.
Thanks guys.
a := fiber.AcquireAgent()
req := a.Request()
req.Header.SetMethod(fiber.MethodGet)
req.SetRequestURI(fmt.Sprintf(formatUrl, args...))
if err := a.Parse(); err != nil {
h.Logger.Error("%v", err)
return fiber.StatusInternalServerError, nil, []error{err}
}
customDialer := fasthttp.TCPDialer{
Concurrency: 1000,
LocalAddr: &net.TCPAddr{
IP: h.IPPool[atomic.AddUint32(&h.IpIdx, 1)%uint32(len(h.IPPool))],
},
}
a.HostClient.Dial = func(addr string) (net.Conn, error) {
return customDialer.Dial(addr)
}
Creating a custom dialer and dial func allows you to change the local address associated with the http request.

Fasthttp error when reading request headers: invalid header key " http/1.1\r\nuser-Agent"

I am just started learning Go, and this question made me stuck.
Trying to test request handling on localhost in testing func using github.com/valyala/fasthttp.
First running the server like in https://github.com/valyala/fasthttp/blob/master/server_example_test.go:
ln, err := net.Listen("tcp", ":8080")
if err != nil {
log.Fatalf("error in net.Listen: %s", err)
}
requestHandler := func(ctx *fasthttp.RequestCtx) {
fmt.Println(ctx, "Requested path is")
}
if err := fasthttp.Serve(ln, requestHandler); err != nil {
log.Fatalf("error in Serve: %s", err)
}
then if I run the request func (FastRequest(url string)) from the same testing function it works fine...
Fasthttp request func:
func FastRequest(url string) error {
Request := &fasthttp.Request{}
Response := &fasthttp.Response{}
FastHTTPClient := &fasthttp.Client{}
Request.SetRequestURI(url)
for {
err := FastHTTPClient.DoTimeout(Request, Response, time.Minute)
switch err {
case fasthttp.ErrTimeout, fasthttp.ErrDialTimeout:
<-time.After(time.Minute * 2)
continue
case fasthttp.ErrNoFreeConns:
<-time.After(time.Minute * 2)
continue
case nil:
return nil
default:
if strings.Contains(err.Error(), "connection reset by peer") {
<-time.After(time.Minute * 2)
continue
} else {
return err
}
}
}
}
But what I truly need to test is sending a request from my object, which implements the same FastRequest method in goroutine.
And here I've got this error message:
error when serving connection ":8080"<->":51325": error when reading request headers: invalid header key " http/1.1\r\nuser-Agent". Buffer size=206, contents: "GET here_is_request_url \n http/1.1\r\nuser-Agent: fasthttp\r\nHost: localhost:8080\r\n\r\n"
In FastRequest I haven't specified any user agent and the functions FastRequest() are the same. Only the place where the function is called is different. Whether it's called in goroutine or not does not matter.
So, fasthttp.RequestCtx cannot parse its own header? or what is going on?
==========================================================================
Also, I should have added that in first case I've used fasthttp v1.6.0, when I changed it to 1.8.0 the error was:
error when serving connection ":8080"<->":57093": error when reading request headers: invalid header name. Buffer size=215, contents: "GET here_is_request_url\n HTTP/1.1\r\nUser-Agent: fasthttp\r\nHost: localhost:8080\r\n\r\n"
And finally, the issue was in "/n" added at the end of the url, that works for real servers, by my small localhost server couldn't handle it.
... contents: "GET here_is_request_url \n http/1.1\r\n ...
^^^^^^^^^^^^^
WRONG!
The URL you use in your code likely has a newline character (\n) still at the end since this is included in your request before the HTTP version and thus messes up the HTTP request. A real URL should have no white space which includes spaces and also newline characters.
Additionally the HTTP version should be all-uppercase HTTP/1.1, i.e. your lower-case http/1.1 is wrong too. You don't show how you create the HTTP request but it is very likely messed up.

Go httputil.ReverseProxy not overriding the Host header

I'm basically trying to write a reverse proxy server so that when I curl localhost:8080/get it proxies the request to https://nghttp2.org/httpbin/get.
Note: the https://nghttp2.org/httpbin/get service listed above is http/2. But this behavior happens with http/1 as well, such as https://httpbin.org/get.
I'm using httputil.ReverseProxy for this and I'm rewriting the URL while customizing the Host header to not to leak the localhost:8080 to the actual backend.
However, the request still hits the backend with Host: localhost:8080 no matter how many times I set it on the header. Similarly, I used mitmproxy to snoop on the request and it looks like the net/http.Client sets the :authority pseudo-header to localhost:8080
Here's my source code:
package main
import (
"log"
"net/http"
"net/http/httputil"
)
func main() {
proxy := &httputil.ReverseProxy{
Transport: roundTripper(rt),
Director: func(req *http.Request) {
req.URL.Scheme = "https"
req.URL.Host = "nghttp2.org"
req.URL.Path = "/httpbin" + req.URL.Path
req.Header.Set("Host", "nghttp2.org") // <--- I set it here first
},
}
log.Fatal(http.ListenAndServe(":8080", proxy))
}
func rt(req *http.Request) (*http.Response, error) {
log.Printf("request received. url=%s", req.URL)
req.Header.Set("Host", "nghttp2.org") // <--- I set it here as well
defer log.Printf("request complete. url=%s", req.URL)
return http.DefaultTransport.RoundTrip(req)
}
// roundTripper makes func signature a http.RoundTripper
type roundTripper func(*http.Request) (*http.Response, error)
func (f roundTripper) RoundTrip(req *http.Request) (*http.Response, error) { return f(req) }
When I query curl localhost:8080/get the request gets proxied to https://nghttp2.org/httpbin/get. The echoed response shows that clearly my directives setting the Host header didn't do anything:
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "localhost:8080",
"User-Agent": "curl/7.54.0"
},
"origin": "2601:602:9c02:16c2:fca3:aaab:3914:4a71",
"url": "https://localhost:8080/httpbin/get"
}
mitmproxy snooping also clearly shows that the request was made with :authority pseudo-header set to localhost:8080:
From http.Request docs:
// For server requests, Host specifies the host on which the URL
// is sought. Per RFC 7230, section 5.4, this is either the value
// of the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
// To prevent DNS rebinding attacks, server Handlers should
// validate that the Host header has a value for which the
// Handler considers itself authoritative. The included
// ServeMux supports patterns registered to particular host
// names and thus protects its registered Handlers.
//
// For client requests, Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
So the value of URL.Host is only used in case request.Host is empty which is not the case. Setting request.Host should resolve the issue:
req.Host = "nghttp2.org"
Related issue discussed here.

How can I get the client IP address and user-agent in Golang gRPC?

I set up a series of gRPC requests and responses which all work fine, but I'm stuck when I try to get the client IP address and user-agent who is calling my gRPC APIs.
I read the Go gRPC documentation and other sources, but didn't find much valuable information. Few of them are talking about gRPC in Golang.
Should I set up a key-value to store the IP address in the context when setting up the gRPC APIs?
In Golang GRPC, you can use
func (UserServicesServer) Login(ctx context.Context, request *sso.LoginRequest) (*sso.LoginResponse, error) {
p, _ := peer.FromContext(ctx)
request.Frontendip = p.Addr.String()
.
.
}
But, do not forget import "google.golang.org/grpc/peer"
For grpc-gateway is used, the client IP address may be retrieved through x-forwarded-for like this:
// Get IP from GRPC context
func GetIP(ctx context.Context) string {
if headers, ok := metadata.FromIncomingContext(ctx); ok {
xForwardFor := headers.Get("x-forwarded-for")
if len(xForwardFor) > 0 && xForwardFor[0] != "" {
ips := strings.Split(xForwardFor[0], ",")
if len(ips) > 0 {
clientIp := ips[0]
return clientIp
}
}
}
return ""
}
In Golang GRPC, context has 3 values
authority
content-type
user-agent
md,ok:=metadata.FromIncomingContext(ctx)
fmt.Printf("%+v%+v",md,ok)

How to get remote client's IPV4 address from request object

I am using go-gin as server and rendering an html using the code like the following
func dashboardHandler(c *gin.Context) {
c.HTML(200, "dashboard", gin.H{
"title": "Dashboard"
})
Along with title I want to pass the remote client's IPV4 address as well. I tried using the following code to get the IP address but for localhost it gives me ::1:56797 as output. My server is running on localhost:8080
ip, port, err := net.SplitHostPort(c.Request.RemoteAddr)
fmt.Println(ip + ":" + port)
if err != nil {
fmt.Println(err.Error())
}
I followed Correct way of getting Client's IP Addresses from http.Request (Golang) for reference. Is there any way I get the IPV4 address from the request?
you can use this function to get the ip and user agent, but it will give a bracket character if you are trying from localhost but if you try from somewhere else it will work.
func GetIPAndUserAgent(r *http.Request) (ip string, user_agent string) {
ip = r.Header.Get("X-Forwarded-For")
if ip == "" {
ip = strings.Split(r.RemoteAddr, ":")[0]
}
user_agent = r.UserAgent()
return ip, user_agent
}

Resources