I'm new to golang and I'm trying to build a small local proxy. The request kinda works from Postman -> localhost:9097 -> localhost:9098 and back. But the content-length is 120 and the response body is just gibberish:
I expect to get a json body like { "result": { "id": "1", "price": 100, "quantity": 1 } }
If a make a request directly to :9098 I see that the response header transfer-encoding is chunked. Any idea how to adjust my code to parse the response body from the server properly and send it back to the client?
func httpHandler(w http.ResponseWriter, req *http.Request) {
reqURL := fmt.Sprint(req.URL)
newUrl = "http://localhost:9098" + reqURL
//forward request
client := http.Client{}
freq, reqerror := http.NewRequest(req.Method, newUrl, nil)
if reqerror != nil {
log.Fatalln(reqerror)
}
freq.Header = req.Header
freq.Body = req.Body
resp, resperr := client.Do(freq)
if resperr != nil {
log.Println(resperr)
fmt.Fprintf(w, "Error. No response")
return
}
defer resp.Body.Close()
body, ioerr := io.ReadAll(resp.Body)
if ioerr != nil {
log.Println(ioerr)
fmt.Fprintf(w, "IO Error (Response body)")
return
}
w.Header().Set("Content-Type", resp.Header.Get("Content-Type"))
w.WriteHeader(resp.StatusCode)
fmt.Fprintf(w, string(body))
}
Managed to solve this now! Thanks to Steffen Ullrich for pointing out the the issue could be "about compressed content".
Removing the Accept-Encoding header as mentioned here worked like a charm.
...
// if you manually set the Accept-Encoding request header, than gzipped response will not automatically decompressed
req.Header.Del("Accept-Encoding")
freq.Header = req.Header
...
Related
I just started to use Golang and I want to remake my already working NodeJS/TypeScript app in Go.
One endpoint of my API simply adds server-side generated authorization headers and sends a request to a remote API. Basically filling those headers for me by calling my API instead of the remote API.
This is what I am currently writing
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-Id", "---------")
client := &http.Client{}
res, err := client.Do(req)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
var forwardedBody interface{}
json.Unmarshal(body, &forwardedBody)
return ctx.Status(fiber.StatusOK).JSON(forwardedBody)
}
I'd like to know if I am on the right steps, because making a request, parsing the JSON response with ioutil then unmarshall it to send it back seems kind of overboard for the simplicity of what I am trying to achieve ?
Edit: Thank you for the help, this is what I will be going for
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-ID", "---------")
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return ctx.SendStatus(fiber.StatusBadRequest)
}
ctx.Set("Content-Type", "application/json; charset=utf-8")
return ctx.Status(res.StatusCode).SendStream(res.Body)
}
You can use httputil.ReverseProxy. Which takes a base URL and forwards requests to the base URL, concatenating the path.
ReverseProxy is an HTTP Handler that takes an incoming request and sends it to another server, proxying the response back to the client.
http.Handle("/", &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme = "https"
r.URL.Host = "go.dev"
r.Host = r.URL.Host
r.Header.Set("X-Foo", "Bar")
},
})
If you are not serving this from the root path / you can use StripPrefix.
http.HandleFunc("/foo/", http.StripPrefix("/foo/", proxy)
There is also a helper function NewSingleHostReverseProxy, which possibly removes the need to configure the proxy struct yourself. But I think it will be better to set the Host header along with your custom header.
You don't need to attempt to parse the data as JSON. This will be problematic if any of your endpoints don't return JSON, anyway, so just inject the body directly into the response:
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
// Inject the body from the inner response into the actual response so it can be returned
ctx.Response().SetBody(body)
return cx.Status(fiber.StatusOK)
Does golang's net/http package support requests with chunked transfer-encoding? Thus far I have been able to use the Hijacker interface (https://golang.org/src/net/http/server.go?s=6173:6875#L156) to at least not close the connection and receive the full chunked request, but not yet parsing the chunks and suspect I may be going down the wrong path with this.
From https://golang.org/src/net/http/httputil/httputil.go?s=688:732#L10, I see there is a chunked reader, but appears to be for internal use.
Essentially, I'm trying to accept an HTTP PUT with 'chunked' transfer-encoding and send it off to a backend server 'on-the-fly' (i.e. without buffering the full request in golang). I have no control over the upstream request. Is there a recommended way to handle such a request, or is Hijacker the way to do it?
The net/http client and server transparently read and write chunked bodies.
To accept a chunked request and send it to another HTTP server, pass the server request body as the client request body. Here's now to forward the body to another server as a PUT:
func handler(w http.ResponseWriter, r *http.Request) {
creq, err := http.NewRequest("PUT", url, r.Body)
if err != nil {
// handle error
}
if ct := r.Header.Get("Content-Type"); ct != "" {
creq.Header.Set("Content-Type", ct)
}
cresp, err := http.DefaultClient.Do(creq)
if err != nil {
// handle error
}
... do something with cresp.
}
If you want to copy to a file, then io.Copy the request body to the file.
func handler(w http.ResponseWriter, r *http.Request) {
f, err := os.Create("fname")
if err != nil {
// handle error
}
_, err := io.Copy(f, r.Body)
if err != nil {
// handle error
}
...
}
These snippets copy the body 'on the fly'.
I'm attempting to use Go to write a utility that authenticates and uploads a file by making a multipart http request to our server. Everything seems to go okay, except the file is not arriving on the server. Looking at it further it appears the multipart in the request is empty. Code and request output below. What am I missing in my Go code?
The Code: (I've changed the URL...)
package main
import (
"net/http"
"mime/multipart"
"strings"
"fmt"
"io/ioutil"
"io"
"os"
"bytes"
"flag"
"encoding/json"
)
var (
filename = flag.String("filename", "", "file to upload")
name = flag.String("name", "", "name to give file on server")
username = flag.String("username", "", "username for authentication")
password = flag.String("password", "", "password for authentication")
)
func main() {
flag.Parse()
// Create multipart
var b bytes.Buffer
w := multipart.NewWriter(&b)
f, _ := os.Open(*filename) //open file to send
defer f.Close()
fw, err := w.CreateFormFile("file", *name) //give file a name
if err != nil {
fmt.Println(err)
}
if _, err := io.Copy(fw, f); err != nil { //copy the file to the multipart buffer
fmt.Println(err)
}
w.Close()
// print the head of the multipart data
bs := b.Bytes()
fmt.Printf("%+v\n\n", string(bs[:1000]))
// Send authentication/login
r, e := http.Post("https://mysite/login", "application/json", strings.NewReader(fmt.Sprintf("{\"username\":\"%s\",\"password\":\"%s\"}", *username, *password)))
if e != nil {
fmt.Println(e)
} else {
// Get the token from the body
type Body struct {
Token string
}
// convert json to get the token
body, _ := ioutil.ReadAll(r.Body)
bd := bytes.NewBuffer(body)
dec := json.NewDecoder(bd)
var m Body
dec.Decode(&m)
// Upload file
req, err := http.NewRequest("POST", "https://mysite/api/apps", &b)
if err != nil {
fmt.Printf("%v\n", err)
}
req.Header.Set("Authentication", fmt.Sprintf("Bearer: %s", m.Token))
req.Header.Set("Content-Type", w.FormDataContentType())
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
fmt.Printf("%v\n", err)
}
// print status and request body
fmt.Println(res.Status)
fmt.Printf("%+v\n", res.Request)
}
}
The first thing I print is the byte buffer, b, which contains the multipart data, everything looks good from here. (it was an xml file)
--83451b003d8e5cc38c0e8f60ad318e522cad4818cf293745c84ec36d26d5
Content-Disposition: form-data; name="file"; filename="snapshot-162224-820-99"
Content-Type: application/octet-stream
<manifest>
<projects>
<project name=........
Next I print the status of the request.
200 OK
Then I printed out the request structure, here is where I saw the MultipartForm was empty.
&{Method:GET URL:https://mysite/home/ Proto: ProtoMajor:0 ProtoMinor:0 Header:map[Authentication:[Bearer: DY0LCJL0g] Content-Type:[multipart/form-data; boundary=83451b003d8e5cc38c0e8f60ad318e522cad4818cf293745c84ec36d26d5] Referer:[http://mysite/home/]] Body:<nil> GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host: Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:0xc42018a360 ctx:<nil>}
I highly doubt if the server really received nothing. The behavior of the printed body having a nil body is expected and documented in http.Response
// Request is the request that was sent to obtain this Response.
// Request's Body is nil (having already been consumed).
// This is only populated for Client requests.
Request *Request
If you want to debug the request body sented, you should either use a mock server or a proxy.
On another note, your code's attempt to login is not going to work. It does not maintain the cookie of login info, so the later requests cannot utilize them.
I am using net/http library in 'Go' to make an HTTP GET request. In the response, i get 12 headers. But when i run the exact same query through postman, i get 16 headers. One of those missing is 'Content-Encoding'. I understand this must be a CORS issue.
But since i have not set the header Accept-Encoding: gzip in my request, and i am still getting the gzip encoding in response, the Go transport is not automatically decompressing the response for me. So, i need to be able to manually detect the encoding and then decompress it. But, i cannot detect if the 'Content-Encoding' header is missing in the response.
Here is my code where i try to do this:
func calcDistanceAndDurationWithUberApi(originLat float64, originLon float64, destinationLat float64, destinationLon float64) (float64, float64, error) {
endpoint := "https://api.uber.com/v1.2/estimates/price"
parameters := fmt.Sprintf("?start_latitude=%v&start_longitude=%v&end_latitude=%v&end_longitude=%v", originLat, originLon, destinationLat, destinationLon)
req, err := http.NewRequest("GET", endpoint + parameters, nil)
if err != nil {
return 0, 0, err
}
req.Header.Add("Authorization", "Token " + getUberApiKey())
req.Header.Add("Accept-Language", "en_US")
req.Header.Add("Content-Type", "application/json")
httpClient := &http.Client{}
resp, err := httpClient.Do(req)
if err != nil {
return 0, 0, err
}
if resp.StatusCode != 200 {
return 0, 0, errors.NotFound("Response: %v", resp.StatusCode)
}
defer resp.Body.Close()
pretty.Println("- REQUEST: ")
pretty.Println(req)
// Check if server sent gzipped response. Decompress if yes.
var respReader io.ReadCloser
switch resp.Header.Get("Content-Encoding") {
case "gzip":
fmt.Println("Content-Encoding is gzip")
respReader, err = gzip.NewReader(resp.Body)
defer respReader.Close()
default:
fmt.Println("Content-Encoding is Not gzip")
respReader = resp.Body
}
pretty.Println("- RESPONSE HEADER: ")
pretty.Println(resp.Header)
pretty.Println("- RESPONSE BODY: ")
pretty.Println(respReader)
return 0, 0, nil
}
The response status is '200 OK'. Here is the output (Response):
- RESPONSE HEADER:
http.Header{
"Content-Language": {"en"},
"Cache-Control": {"max-age=0"},
"X-Uber-App": {"uberex-nonsandbox", "optimus"},
"Strict-Transport-Security": {"max-age=604800", "max-age=2592000"},
"X-Content-Type-Options": {"nosniff"},
"Date": {"Fri, 19 May 2017 07:52:17 GMT"},
"Content-Geo-System": {"wgs-84"},
"Connection": {"keep-alive"},
"X-Frame-Options": {"SAMEORIGIN"},
"X-Xss-Protection": {"1; mode=block"},
"Server": {"nginx"},
"Content-Type": {"application/json"},
}
- RESPONSE BODY:
&http.gzipReader{
body: &http.bodyEOFSignal{
body: &http.body{
src: &internal.chunkedReader{
r: &bufio.Reader{
buf: {0x48, 0x54, .......... }
I gave in to the stubbornness of the uber api and added another request header, req.Header.Add("Accept-Encoding", "gzip").
Now i am getting the response header "Content-Encoding": "gzip", although i am still getting an undecipherable response body, but that's beyond the scope of this question.
If you don't disable compression [1], and you don't manually request compression with Accept-Encoding: gzip, then what I call "automatic mode" is used. With automatic mode, Go automatically adds Accept-Encoding: gzip, then if server responds Content-Encoding: gzip, Go wrap the response body in a Gzip reader, and removes the Content-Encoding and Content-Length response headers [2]. I disagree with this practice, as the end user is essentially being lied to about what the true response was. Contrast this with cURL, which gives you the pure response, regardless of what you do:
PS C:\> curl -v --compressed https://github.com/manifest.json
< content-encoding: gzip
< content-length: 345
To deal with this, I wrote a wrapper for http.Transport:
package mech
import (
"compress/gzip"
"io"
"net/http"
"strings"
)
type Transport struct { http.Transport }
func (t Transport) RoundTrip(req *http.Request) (*http.Response, error) {
if !t.DisableCompression {
req.Header.Set("Accept-Encoding", "gzip")
}
res, err := t.Transport.RoundTrip(req)
if err != nil {
return nil, err
}
if strings.EqualFold(res.Header.Get("Content-Encoding"), "gzip") {
gz, err := gzip.NewReader(res.Body)
if err != nil {
return nil, err
}
res.Body = readCloser{gz, res.Body}
}
return res, nil
}
type readCloser struct {
io.Reader
io.Closer
}
https://golang.org/pkg/net/http#Transport.DisableCompression
https://github.com/golang/go/blob/go1.16.5/src/net/http/transport.go#L2186-L2192
I have a very basic http server and http client. The server is not sending out the data when chunked despite flushing it. I have validated both with a go http client and curl.
Here is the server:
func main() {
r := mux.NewRouter()
baseURL := ""
r.HandleFunc(baseURL+"/",handler).Methods("GET")
listener, _ := net.Listen("tcp", "127.0.0.1:0")
http.Handle("/",r)
go http.Serve(listener,nil)
url = "http://" + listener.Addr().String() + "/"
fmt.Println(url)
TestSend()
}
func writeHeaders(w http.ResponseWriter, code int) {
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(code)
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Println("handler")
writeHeaders(w, 200)
rsc := bufio.NewScanner(strings.NewReader(response))
for rsc.Scan() {
sender <- rsc.Text()
}
for d := range sender {
w.Write([]byte(d))
w.(http.Flusher).Flush()
fmt.Println("wrote and flushed "+d)
}
}
And the client:
res, _ := http.Get(url)
reader := bufio.NewReader(res.Body)
for {
fmt.Println("ReadBytes")
data, err := reader.ReadBytes('\r')
fmt.Println("read")
if err != nil {
fmt.Println("error!")
fmt.Println(err)
return
}
fmt.Println("read line "+string(data))
}
I get the server saying it is sending and flushing, and I get the "ReadBytes" saying it is waiting to read, but it never reads any lines:
$ go run httptest.go
http://127.0.0.1:55967/
handler
wrote and flushed abc
wrote and flushed def
wrote and flushed ghi
wrote and flushed jkl
wrote and flushed mno
ReadBytes
(here it just waits)
If, while it is running, I curl, I get the right headers, but no data:
$ curl -i http://127.0.0.1:55957/
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sun, 02 Aug 2015 07:01:21 GMT
Transfer-Encoding: chunked
(no data here at all)
Clearly the data is not leaving the server, but why? I am Flushing it.