I have a REST endpoint provided by a Java Spring Boot app that I am trying to hit with a simple Go (go1.8 darwin/amd64) program. I have tried to hit it in 2 separate ways:
const (
LOCAL_BASE = "http://localhost:11400"
ADD_FUNDS_ENDPOINT = "/sub-accounts/%d/add-funds"
)
type InputArgumentsJson struct {
Amount float64 `json:"amount"`
BufferAmount float64 `json:"bufferAmount"`
FromSubAccountID int `json:"fromSubAccountId"`
}
...
// set up json body
inputArguments := &InputArgumentsJson{Amount: 1100, BufferAmount: 0,
FromSubAccountID: spendFundId}
bytes := new(bytes.Buffer)
json.NewEncoder(bytes).Encode(inputArguments)
// set up the url to call
endpoint := fmt.Sprintf(ADD_FUNDS_ENDPOINT, billSetAsideId)
url := LOCAL_BASE + endpoint
client := &http.Client{}
req, err := http.NewRequest("POST", url, bytes)
if err != nil {
fmt.Println("ERROR WHILE BUILDING REQUEST")
fmt.Println(err)
}
req.Header.Add("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
fmt.Println("ERROR WHILE MAKING REQUEST")
fmt.Println(err)
}
fmt.Println(resp)
and:
resp, err := http.Post(url, "application/json; charset=utf-8", bytes)
if err != nil {
fmt.Println("ERROR WHILE MAKING REQUEST")
fmt.Println(err)
}
fmt.Println(resp)
I added a print out to see if the URL and JSON body were right and they appeared to be:
fmt.Printf("\nMaking request to %s with:\n %s\n", url, bytes)
printed out
Making request to http://localhost:11400/sub-accounts/167/add-funds with:
{"amount":1100,"bufferAmount":0,"fromSubAccountId":166}
but I am getting a 403 Forbidden when running it.
I can hit this endpoint without fail using both Postman and the following Curl command:
curl -H "Content-Type: application/json" -X POST -d '{"amount":1100,"bufferAmount":0,"fromSubAccountId":166}' http://localhost:11400/sub-accounts/167/add-funds
Does anyone have any thoughts on what I may be missing here? To add further to the intrigue I can successfully hit a different GET endpoint from the Go program.
Thanks in advance!
Edited to show that I was catching the errors and the surrounding logic for creating the URL and JSON body
You did not show what bytes is but that's most probably the problem: both NewRequest and Post take an io.Reader and read the body from that reader. This reading typically can happen only once (think of a stream). When you fmt.Printf("%s") the io.Reader this printing will consume the content (most probably by invoking a String() method) and the actual request is done with an empty body.
Always show whole code. And do not try to read twice from an io.Reader.
I assume there are 2 Go apps and runs on a different host, which results a CORS problem. Consider adding the Access-Control-Allow-Origin at your service with your client's host.
w.Header().Set("Access-Control-Allow-Origin", "*")
You can change * with specified hostname which your client application runs.
OMG....I just found the problem. It turned out that it was failing (as I was expecting), but it was erroneously returning it as a 403 rather than a 400. Ugh...sometimes when you stare at something for a long time the obvious things completely escape notice.
you should set User-Agent in request header
req, _ := http.NewRequest(http.MethodGet, urlString, nil)
req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36")
Related
I am trying to test the following line of code:
httpReq.Header.Set("Content-Type", "application/json")
I am mocking the request to an external api in this way:
httpmock.RegisterResponder(http.MethodPost, "do-not-exist.com",
httpmock.NewStringResponder(http.StatusOK, `{
"data":{"Random": "Stuff"}}`),
)
And want to test if the request to the api has the header that I assigned. Is there a way I could achieve this?
With the help of the comment by #Kelsnare I was able to solve this issue in the following way:
httpmock.RegisterResponder(http.MethodPost, "do-not-exist.com",
func(req *http.Request) (*http.Response, error) {
require.Equal(t, req.Header.Get("Content-Type"), "application/json")
resp, _ := httpmock.NewStringResponder(http.StatusOK, `{
"data":{"Random": "Stuff"}}`)(req)
return resp, nil},
)
I wrote my own func of http.Responder type and used httpmock.NewStringResponder inside that func.
response_test.go illustrates how the header is tested:
response, err := NewJsonResponse(200, test.body)
if err != nil {
t.Errorf("#%d NewJsonResponse failed: %s", i, err)
continue
}
if response.StatusCode != 200 {
t.Errorf("#%d response status mismatch: %d ≠ 200", i, response.StatusCode)
continue
}
if response.Header.Get("Content-Type") != "application/json" {
t.Errorf("#%d response Content-Type mismatch: %s ≠ application/json",
i, response.Header.Get("Content-Type"))
continue
You can see an example of table-driven test with httpmock.RegisterResponder here.
I'm writing some middleware and I need to be able to log the response body content even when the destination is using TLS encryption.
I have a handler chain within which I store the response body in an intermediate buffer, so that I can read it more than once. This is based on the excellent example provided by icza (Golang read request body).
In my handler func, I'm doing this....
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
// Print the response body to stdout
fmt.Printf("Dest HTTP response body: %s\n", body)
bRdr := bytes.NewReader(body)
n, err := io.Copy(w, bRdr) // Copy the entire response body into our outgoing response
What I'm finding is that I get readable output when connection to a destination not using TLS, but when connected to a destination using TLS, it seems the response body is still encrypted, though the Copy into the final response to the originator results in the originator receiving valid response body content.
Is this the expected behaviour for reads of the response body with an encrypted path?
Can I decrypt this data to be able make it readable? I've read the http, tls and crypto package documentation, but have not found any clues.
I'm not sure if I understand the problem but here is me calling an https google link and printing the output.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"golang.org/x/net/http2"
)
func main() {
client := &http.Client{Transport: transport2()}
res, err := client.Get("https://www.google.com")
if err != nil {
log.Fatal(err)
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
res.Body.Close()
fmt.Printf("Code: %d\n", res.StatusCode)
fmt.Printf("Body: %s\n", body)
}
func transport2() *http2.Transport {
return &http2.Transport{
DisableCompression: true,
AllowHTTP: false,
}
}
Thanks all for your comments. Travis seems to have identified the issue I'm having. It appears the response body I'm reading is gzip encoded (the response contains "Content-Encoding: gzip"). In order to verify that this was the case, I had to explicitly remove the "Accept-Encoding: gzip" header that was in the originating request before forwarding and also configure the Transport to set "DisableCompression: true". Once I made both of those changes, I then see responses with no "Content-Encoding" header and the body I read is human readable.
I'm looking for an efficient way to read HTTP headers from a textfile to be later sent with an HTTP request. Consider the following code (which currently contains basic net/http request functionality):
func MakeRequest(target string, method string) {
client := &http.Client{}
req, _ := http.NewRequest(method, target, nil)
//Headers manually..
req.Header.Add("If-None-Match", `some value`)
response, _ := client.Do(req)
body, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(body))
}
I started by using ioutil.ReadFile like this:
func main() {
data, _ := ioutil.ReadFile("/opt/tests/req.txt")
fmt.Print(string(data))
}
But taking this text, splitting it by some indicator (lets say ":") and then placing the information in req.Header.Add("var1", "var2") per-header seems like an over-kill.
Question: Any better way to send HTTP requests with headers from a text file in go?
net/http has a method ReadRequest which can create a new Request object from a bufio.Reader. Assuming that your file contains a real HTTP request (instead of only the part of the request which consists of lines with key: value) all you need to do is create a new bufio.Reader from the file, i.e. like this (error handling omitted):
rdr,_ := os.Open("req.txt")
req,_ := http.ReadRequest(bufio.NewReader(rdr))
fmt.Printf("%+v\n", req)
If you only want some headers defined, another option is to define the headers in a Json file and apply the following code (file reading not included):
var jsonMap map[string]string
err = json.Unmarshal(jsonBytesFromFile, &jsonMap)
if err != nil {
log.Fatal("unable to parse json: ", err)
}
for k, v := range jsonMap {
log.Printf("setting Header : %s : %s", k, v)
responseWriter.Header().Add(k, v) // you may prefer Set()
}
The json looks like this:
{
"Content-type": "text/plain",
"Cache-Control": "only-if-cached"
}
I have the following code:
package main
import (
"bytes"
"fmt"
"net/http"
)
func main() {
var jsonStr = []byte(`{"title":"Buy cheese and bread for breakfast."}`)
req, err := http.NewRequest("POST", "/", bytes.NewBuffer(jsonStr))
if err != nil {
panic(err)
}
req.Header.Set("X-Custom-Header", "myvalue")
req.Header.Set("Content-Type", "application/json")
req.ParseForm()
fmt.Printf("%v:%v", "title", req.Form.Get("title"))
}
I am unable to extract the "title" param and not sure why.
As noted in the GoDoc for the http.Request.ParseForm method, the type of the body must be application/x-www-form-urlencoded, not JSON like your current example:
For other HTTP methods, or when the Content-Type is not application/x-www-form-urlencoded, the request Body is not read, and r.PostForm is initialized to a non-nil, empty value.
Here is an updated example of your code using a form body, which gives the intended result: https://play.golang.org/p/Zrw05T2Zb5Z
If you want to extract values from a JSON body, that can be done using a method such as json.Unmarshal, however a JSON body doesn't represent a form.
The 3rd argument of http.NewRequest is the http payload.
In your case, payload type is application/json. It's need to be treated as json, only then you'll be able to get certain value from the it. In this case, we just cannot use the same technique like on getting value from query string or form data.
So just unmarshal the jsonStr data into map or struct.
res := make(map[string]interface{})
err := json.Unmarshal(jsonStr, &res)
if err != nil {
panic(err)
}
fmt.Printf("%#v \n", res["title"])
To be honest I'm quite confused with your question, why you need to get the payload from http client request.
If what you want is actually how to get the payload from the web server end, you can get it by decoding the request body. Example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
payload := make(map[string]interface{})
err := json.NewDecoder(r.Body).Decode(&payload)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
title := payload["title"].(string)
w.Write([]byte(title))
})
Curl example (based on your code):
curl -d '{"title":"Buy cheese and bread for breakfast."}' \
-H "Content-Type: application/json" \
-X POST http://localhost:9000
Output:
Buy cheese and bread for breakfast.
Because your request isn't a form. It doesn't have any GET parameters, and it isn't form-encoded data.
For other HTTP methods, or when the Content-Type is not application/x-www-form-urlencoded, the request Body is not read, and r.PostForm is initialized to a non-nil, empty value.
[ https://golang.org/pkg/net/http/#Request.ParseForm ]
You're free to parse the body of the request as application/json, but that isn't the same as form data.
My code get file from remote url and download file in browser:
func Index(w http.ResponseWriter, r *http.Request) {
url := "http://upload.wikimedia.org/wikipedia/en/b/bc/Wiki.png"
...
resp, err := client.Get(url)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(len(body))
//download the file in browser
}
func main() {
http.HandleFunc("/", Index)
err := http.ListenAndServe(":8000", nil)
if err != nil {
fmt.Println(err)
}
}
code: http://play.golang.org/p/x-EyR2zFjv
Get file is ok, but how to downloaded it in browser?
To make the browser open the download dialog, add a Content-Disposition and Content-Type headers to the response:
w.Header().Set("Content-Disposition", "attachment; filename=WHATEVER_YOU_WANT")
w.Header().Set("Content-Type", r.Header.Get("Content-Type"))
Do this BEFORE sending the content to the client. You might also want to copy the Content-Length header of the response to the client, to show proper progress.
To stream the response body to the client without fully loading it into memory (for big files this is important) - simply copy the body reader to the response writer:
io.Copy(w, resp.Body)
io.Copy is a nice little function that take a reader interface and writer interface, reads data from one and writes it to the other. Very useful for this kind of stuff!
I've modified your code to do this: http://play.golang.org/p/v9IAu2Xu3_
In case you already have the file on disk, just use http.ServeFile(). It automatically handles Content-Length so that the browser can display a download progress.
w.Header().Set("Content-Disposition", "attachment; filename="+strconv.Quote(filename))
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeFile(w, r, filePath)
What you need to use is the Content-Disposition header with attachment value. MDN web docs says the following about the header:
[...], the Content-Disposition response header is a header indicating if the content is expected to be displayed inline in the browser, that is, as a Web page or as part of a Web page, or as an attachment, that is downloaded and saved locally.
If you want to specify a filename for the clients you can use the filename directive. In the following format:
Content-Disposition: attachment; filename="filename.jpg"
This is an optional parameter and it has some restrictions.
[...] The filename is always optional and must not be used blindly by the application: path information should be stripped, and conversion to the server file system rules should be done. [...]
To format filename properly in the header you should use mime.FormatMediaType. Example:
cd := mime.FormatMediaType("attachment", map[string]string{"filename": d.Name()})
w.Header().Set("Content-Disposition", cd)
In this case for content type you can use application/octet-stream because the browser does not have to know the MIME type of the response.
[...] Generic binary data (or binary data whose true type is unknown) is application/octet-stream. [...]
w.Header().Set("Content-Type", "application/octet-stream")
For outputing the content of a. file I would recommend to use http.ServeContent or http.ServeFile because they handle RFC 7233 - Range Requests out of the box. Example:
f, err := fs.Open(name)
// [...]
cd := mime.FormatMediaType("attachment", map[string]string{"filename": d.Name()})
w.Header().Set("Content-Disposition", cd)
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeContent(w, r, d.Name(), d.ModTime(), f)