I've read this so question and the example from this so question
Here's my demo:
package main
import (
"net/http"
)
func main() {
http.HandleFunc("/", handler1)
_ = http.ListenAndServe(":8008", nil)
}
func handler1(writer http.ResponseWriter, request *http.Request) {
hijacker, _ := writer.(http.Hijacker)
conn, buf, _ := hijacker.Hijack()
defer conn.Close()
_, _ = buf.WriteString("hello world")
_ = buf.Flush()
}
I ran the code with go run serv1.go and sent request with curl, but got HTTP/1.1 502 Bad Gateway. Just wondering if the API has been changed or anything I'd missed?
❯ curl -i "http://localhost:8008/"
HTTP/1.1 502 Bad Gateway
Connection: keep-alive
Keep-Alive: timeout=4
Proxy-Connection: keep-alive
Content-Length: 0
Related
I have been wondering if there is already a method to write all of an http/Response into a []byte? I've found responses that note that the body can be converted easily into a []byte by doing ioutil.ReadAll(response.Body), but is there an already-built solution that writes all of the information (including status codes, headers, trailers, etc.)?
The reason I ask is because I wish to transmit this entire response through a socket to a client, and the Write method of the net library requires a byte array.
httputil.DumpResponse is what you need (Also suggested by Adrian). The following code should help:
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"net/http/httputil"
"os"
)
func main() {
// Create a test server
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
// Set Header
w.Header().Set("HEADER_KEY", "HEADER_VALUE")
// Set Response Body
fmt.Fprintln(w, "DUMMY_BODY")
}))
defer server.Close()
// Request to the test server
resp, err := http.Get(server.URL)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer resp.Body.Close()
// DumpResponse takes two parameters: (resp *http.Response, body bool)
// where resp is the pointer to the response object. And body is boolean
// to dump body or not
dump, err := httputil.DumpResponse(resp, true)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
// Dump the response ([]byte)
fmt.Printf("%q", dump)
}
Output:
"HTTP/1.1 200 OK\r\nContent-Length: 11\r\nContent-Type: text/plain; charset=utf-8\r\n
Date: Wed, 18 Nov 2020 17:43:40 GMT\r\n
Header_key: HEADER_VALUE\r\n\r\n
DUMMY_BODY\n"
Problem:
I'm fowarding to a HTTPS address.
I want to see why removing
req.Host = req.URL.Host causes it to fail. Instead of returning {"Code":"OBRI.FR.Request.Invalid","Id":"c37baec213dd1227","Message":"An error happened when parsing the request arguments","Errors":[{"ErrorCode":"UK.OBIE.Header.Missing","Message":"Missing request header 'x-fapi-financial-id' for method parameter of type String","Url":"https://docs.ob.forgerock.financial/errors#UK.OBIE.Header.Missing"}]}
it returns a 404.
I want to trace the call the proxy returned from
httputil. NewSingleHostReverseProxy is making when I uncommment the line req.Host = req.URL.Host.
Given a request as such:
$ curl http://localhost:8989/open-banking/v2.0/accounts
And the code below (main.go):
package main
import (
"log"
"net/http"
"net/http/httputil"
"net/url"
)
func main() {
target, err := url.Parse("https://rs.aspsp.ob.forgerock.financial:443")
log.Printf("forwarding to -> %s%s\n", target.Scheme, target.Host)
if err != nil {
log.Fatal(err)
}
proxy := httputil.NewSingleHostReverseProxy(target)
http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
// https://stackoverflow.com/questions/38016477/reverse-proxy-does-not-work
// https://forum.golangbridge.org/t/explain-how-reverse-proxy-work/6492/7
// https://stackoverflow.com/questions/34745654/golang-reverseproxy-with-apache2-sni-hostname-error
req.Host = req.URL.Host // if you remove this line the request will fail... I want to debug why.
proxy.ServeHTTP(w, req)
})
err = http.ListenAndServe(":8989", nil)
if err != nil {
panic(err)
}
}
Set the proxy.Transport field to an implementation that dumps the request before delegating to the default transport:
package main
import (
"fmt"
"log"
"net/http"
"net/http/httputil"
"net/url"
)
type DebugTransport struct{}
func (DebugTransport) RoundTrip(r *http.Request) (*http.Response, error) {
b, err := httputil.DumpRequestOut(r, false)
if err != nil {
return nil, err
}
fmt.Println(string(b))
return http.DefaultTransport.RoundTrip(r)
}
func main() {
target, _ := url.Parse("https://example.com:443")
log.Printf("forwarding to -> %s\n", target)
proxy := httputil.NewSingleHostReverseProxy(target)
proxy.Transport = DebugTransport{}
http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
req.Host = req.URL.Host
proxy.ServeHTTP(w, req)
})
log.Fatal(http.ListenAndServe(":8989", nil))
}
The output from this program looks something like this:
2018/10/26 13:06:35 forwarding to -> https://example.com:443
GET / HTTP/1.1
Host: example.com:443
User-Agent: HTTPie/0.9.4
Accept: */*
Accept-Encoding: gzip, deflate
X-Forwarded-For: 127.0.0.1
Or, after removing the req.Host assignment:
2018/10/26 13:06:54 forwarding to -> https://example.com:443
GET / HTTP/1.1
Host: localhost:8989
User-Agent: HTTPie/0.9.4
Accept: */*
Accept-Encoding: gzip, deflate
X-Forwarded-For: 127.0.0.1
Since the Host header is very often used by webservers to route the request to the correct virtual host or backend server, it makes sense that an unexpected Host header ("localhost:8989" in the example above) causes the server to respond with 404.
Setting the Host header with httputil.ReverseProxy is typically done with the Director function:
target, err := url.Parse("https://example.com:443")
if err != nil {
log.Fatal(err)
}
log.Printf("forwarding to -> %s\n", target)
proxy := httputil.NewSingleHostReverseProxy(target)
d := proxy.Director
proxy.Director = func(r *http.Request) {
d(r) // call default director
r.Host = target.Host // set Host header as expected by target
}
log.Fatal(http.ListenAndServe(":8989", proxy))
I am trying to serve a proxy like this:
package main
import (
"net/http"
)
func main() {
http.ListenAndServe(":8080", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
println(r.Host)
}))
}
and calling it with curl
curl -k -x http://localhost:8080 http://golang.org/
I get golang.org printed out. Why I do not get the proxy hostname localhost? is that a bug or limitation with the http proxy?
Update
To clarify, what I am looking for is something like Nginx server address http://nginx.org/en/docs/http/ngx_http_core_module.html#var_server_addr
I should use LocalAddrContextKey but looks like there is a known bug with setting it https://github.com/golang/go/issues/18686
A workaround is to hijack the http.ResponseWriter ex:
package main
import (
"net/http"
)
func main() {
http.ListenAndServe(":8080", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hij, ok := w.(http.Hijacker)
if !ok {
panic("http server does not support hijacker")
}
clientConn, _, err := hij.Hijack()
if err != nil {
panic(err.Error())
}
println(clientConn.LocalAddr().String())
}))
}
I am using net/http library in 'Go' to make an HTTP GET request. In the response, i get 12 headers. But when i run the exact same query through postman, i get 16 headers. One of those missing is 'Content-Encoding'. I understand this must be a CORS issue.
But since i have not set the header Accept-Encoding: gzip in my request, and i am still getting the gzip encoding in response, the Go transport is not automatically decompressing the response for me. So, i need to be able to manually detect the encoding and then decompress it. But, i cannot detect if the 'Content-Encoding' header is missing in the response.
Here is my code where i try to do this:
func calcDistanceAndDurationWithUberApi(originLat float64, originLon float64, destinationLat float64, destinationLon float64) (float64, float64, error) {
endpoint := "https://api.uber.com/v1.2/estimates/price"
parameters := fmt.Sprintf("?start_latitude=%v&start_longitude=%v&end_latitude=%v&end_longitude=%v", originLat, originLon, destinationLat, destinationLon)
req, err := http.NewRequest("GET", endpoint + parameters, nil)
if err != nil {
return 0, 0, err
}
req.Header.Add("Authorization", "Token " + getUberApiKey())
req.Header.Add("Accept-Language", "en_US")
req.Header.Add("Content-Type", "application/json")
httpClient := &http.Client{}
resp, err := httpClient.Do(req)
if err != nil {
return 0, 0, err
}
if resp.StatusCode != 200 {
return 0, 0, errors.NotFound("Response: %v", resp.StatusCode)
}
defer resp.Body.Close()
pretty.Println("- REQUEST: ")
pretty.Println(req)
// Check if server sent gzipped response. Decompress if yes.
var respReader io.ReadCloser
switch resp.Header.Get("Content-Encoding") {
case "gzip":
fmt.Println("Content-Encoding is gzip")
respReader, err = gzip.NewReader(resp.Body)
defer respReader.Close()
default:
fmt.Println("Content-Encoding is Not gzip")
respReader = resp.Body
}
pretty.Println("- RESPONSE HEADER: ")
pretty.Println(resp.Header)
pretty.Println("- RESPONSE BODY: ")
pretty.Println(respReader)
return 0, 0, nil
}
The response status is '200 OK'. Here is the output (Response):
- RESPONSE HEADER:
http.Header{
"Content-Language": {"en"},
"Cache-Control": {"max-age=0"},
"X-Uber-App": {"uberex-nonsandbox", "optimus"},
"Strict-Transport-Security": {"max-age=604800", "max-age=2592000"},
"X-Content-Type-Options": {"nosniff"},
"Date": {"Fri, 19 May 2017 07:52:17 GMT"},
"Content-Geo-System": {"wgs-84"},
"Connection": {"keep-alive"},
"X-Frame-Options": {"SAMEORIGIN"},
"X-Xss-Protection": {"1; mode=block"},
"Server": {"nginx"},
"Content-Type": {"application/json"},
}
- RESPONSE BODY:
&http.gzipReader{
body: &http.bodyEOFSignal{
body: &http.body{
src: &internal.chunkedReader{
r: &bufio.Reader{
buf: {0x48, 0x54, .......... }
I gave in to the stubbornness of the uber api and added another request header, req.Header.Add("Accept-Encoding", "gzip").
Now i am getting the response header "Content-Encoding": "gzip", although i am still getting an undecipherable response body, but that's beyond the scope of this question.
If you don't disable compression [1], and you don't manually request compression with Accept-Encoding: gzip, then what I call "automatic mode" is used. With automatic mode, Go automatically adds Accept-Encoding: gzip, then if server responds Content-Encoding: gzip, Go wrap the response body in a Gzip reader, and removes the Content-Encoding and Content-Length response headers [2]. I disagree with this practice, as the end user is essentially being lied to about what the true response was. Contrast this with cURL, which gives you the pure response, regardless of what you do:
PS C:\> curl -v --compressed https://github.com/manifest.json
< content-encoding: gzip
< content-length: 345
To deal with this, I wrote a wrapper for http.Transport:
package mech
import (
"compress/gzip"
"io"
"net/http"
"strings"
)
type Transport struct { http.Transport }
func (t Transport) RoundTrip(req *http.Request) (*http.Response, error) {
if !t.DisableCompression {
req.Header.Set("Accept-Encoding", "gzip")
}
res, err := t.Transport.RoundTrip(req)
if err != nil {
return nil, err
}
if strings.EqualFold(res.Header.Get("Content-Encoding"), "gzip") {
gz, err := gzip.NewReader(res.Body)
if err != nil {
return nil, err
}
res.Body = readCloser{gz, res.Body}
}
return res, nil
}
type readCloser struct {
io.Reader
io.Closer
}
https://golang.org/pkg/net/http#Transport.DisableCompression
https://github.com/golang/go/blob/go1.16.5/src/net/http/transport.go#L2186-L2192
I have a very basic http server and http client. The server is not sending out the data when chunked despite flushing it. I have validated both with a go http client and curl.
Here is the server:
func main() {
r := mux.NewRouter()
baseURL := ""
r.HandleFunc(baseURL+"/",handler).Methods("GET")
listener, _ := net.Listen("tcp", "127.0.0.1:0")
http.Handle("/",r)
go http.Serve(listener,nil)
url = "http://" + listener.Addr().String() + "/"
fmt.Println(url)
TestSend()
}
func writeHeaders(w http.ResponseWriter, code int) {
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(code)
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Println("handler")
writeHeaders(w, 200)
rsc := bufio.NewScanner(strings.NewReader(response))
for rsc.Scan() {
sender <- rsc.Text()
}
for d := range sender {
w.Write([]byte(d))
w.(http.Flusher).Flush()
fmt.Println("wrote and flushed "+d)
}
}
And the client:
res, _ := http.Get(url)
reader := bufio.NewReader(res.Body)
for {
fmt.Println("ReadBytes")
data, err := reader.ReadBytes('\r')
fmt.Println("read")
if err != nil {
fmt.Println("error!")
fmt.Println(err)
return
}
fmt.Println("read line "+string(data))
}
I get the server saying it is sending and flushing, and I get the "ReadBytes" saying it is waiting to read, but it never reads any lines:
$ go run httptest.go
http://127.0.0.1:55967/
handler
wrote and flushed abc
wrote and flushed def
wrote and flushed ghi
wrote and flushed jkl
wrote and flushed mno
ReadBytes
(here it just waits)
If, while it is running, I curl, I get the right headers, but no data:
$ curl -i http://127.0.0.1:55957/
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sun, 02 Aug 2015 07:01:21 GMT
Transfer-Encoding: chunked
(no data here at all)
Clearly the data is not leaving the server, but why? I am Flushing it.