I wrote a cookie getter and setter. Now I want to test it, and wrote following test function.
func TestAuthorizationReader(t *testing.T) {
tw := httptest.NewServer(testWriter())
tr := httptest.NewServer(Use(testReader()))
defer tw.Close()
defer tr.Close()
c := &http.Client{}
rs, err := c.Get(tw.URL)
assert.NoError(t, err, "Should not contain any error")
// Assign cookie to client
url, err := rs.Location()
fmt.Print(url)
assert.NoError(t, err, "Should not contain any error")
//c.Jar.SetCookies(url, rs.Cookies())
}
The test fail at the second part, as output message I've got
- FAIL: TestAuthorizationReader (0.05s)
Location: logged_test.go:64
Error: No error is expected but got http: no Location header in response
Messages: Should not contain any error
I can not get the URL location pointer, what do I wrong here?
The Response.Location method returns the value of the Location response header. You would usually only expect to see this header for redirect responses, so it isn't surprising you got this error.
If you want to know the URL used to retrieve a particular response, try rs.Request.URL.String(). Even if the HTTP library followed a redirect to retrieve the document, this will look at the request used for this particular response, which is what you'd be after when determining a cookie's origin.
If you just want the client to keep track of cookies set by requests it processes though, all you should need to do is set the Jar attribute on your client. Something like this:
import "net/http/cookiejar"
...
c := &http.Client{
Jar: cookiejar.New(nil),
}
Now cookies set in earlier responses should be set in future requests to the same origin.
Related
The official Go documentation on the datastore package (client library for the GCP datastore service) has the following code snippet for demonstartion:
type Entity struct {
Value string
}
func main() {
ctx := context.Background()
// Create a datastore client. In a typical application, you would create
// a single client which is reused for every datastore operation.
dsClient, err := datastore.NewClient(ctx, "my-project")
if err != nil {
// Handle error.
}
k := datastore.NameKey("Entity", "stringID", nil)
e := new(Entity)
if err := dsClient.Get(ctx, k, e); err != nil {
// Handle error.
}
old := e.Value
e.Value = "Hello World!"
if _, err := dsClient.Put(ctx, k, e); err != nil {
// Handle error.
}
fmt.Printf("Updated value from %q to %q\n", old, e.Value)
}
As one can see, it states that the datastore.Client should ideally only be instantiated once in an application. Now given that the datastore.NewClient function requires a context.Context object does it mean that it should get instantiated only once per HTTP request or can it safely be instantiated once globally with a context.Background() object?
Each operation requires a context.Context object again (e.g. dsClient.Get(ctx, k, e)) so is that the point where the HTTP request's context should be used?
I'm new to Go and can't really find any online resources which explain something like this very well with real world examples and actual best practice patterns.
You may use any context.Context for the datastore client creation, it may be context.Background(), that's completely fine. Client creation may be lengthy, it may require connecting to a remote server, authenticating, fetching configuration etc. If your use case has limited time, you may pass a context with timeout to abort the operation. Also if creation takes longer than the time you have, you may use a context with cancel and abort the mission at your will. These are just options which you may or may not use. But the "tools" are given via context.Context.
Later when you use the datastore.Client during serving (HTTP) client requests, then using the request's context is reasonable, so if a request gets cancelled, then so will its context, and so will the datastore operation you issue, rightfully, because if the client cannot see the result, then there's no point completing the query. Terminating the query early you might not end up using certain resources (e.g. datastore reads), and you may lower the server's load (by aborting jobs whose result will not be sent back to the client).
Does anyone know how to fix this error?
I use Golang to insert data into elasticsearch, but it seems that there is no data inserted because of this error.
{"error":"Content-Type header [] is not supported","status":406}
I already set the content type. Note that I use elasticsearch 6.4.3
request, err := http.NewRequest("POST", urlSearch, bytes.NewBuffer(query))
request.Close = true
request.Header.Set("Content-Type", "application/json")
Last but not least, I use elastigo package to make requests to elasticsearch.
That's a strange response, as it suggests that this line:
request.Header.Set("Content-Type", "application/json")
Failed to add the value to the key slice. In modern go that does not happen, e.g.
data := []byte(`{"a":1}`)
req, err := http.NewRequest("POST", "", bytes.NewBuffer(data))
if err != nil {
fmt.Println(err)
return
}
req.Header.Set("Foo", "Bar")
fmt.Printf("%v\n", req.Header)
Prints
map[Foo:[Bar]]
See go playground.
Are you using older version of Go that doesn't match that behavior? (I'm on 1.11.2 locally.)
Five suggestions:
(1) Handle the err return value from NewRequest to verify there's no problem there (see example above).
(2) Print the request Header value before send to verify it looks right at that point (see example above).
(3) Try the Add method for the Content-Type header instead of Set as an alternative:
func (h Header) Add(key, value string)
(4) Verify that you're not going through a proxy that strips header values.
(5) Verify that "application/json" is an acceptable content type for the endpoint you're hitting, as the empty value in the error response could be erroneous itself.
Good luck!
Creating a byte array of a http request and then trying to read it into a http.request doesn't seem to work when the request includes a body.
req, _ := http.NewRequest(http.MethodPost, "/Bar", strings.NewReader("Foo"))
rReq, _ := httputil.DumpRequest(req, true)
req2, _ := http.ReadRequest(bufio.NewReader(bytes.NewReader(rReq)))
b, _ := ioutil.ReadAll(req2.Body)
fmt.Println(b)
b is an empty array.
Two things are wrong in your code:
You must handle the errors. This would have helpesd you see that you never construct a valid request ("/Bar" is not a valid URL).
Use httputil.DumpRequestOut for outgoing request.
Takeaways: Always handle all errors and always read the whole whole package doc.
Currently my web app is just a router and handlers.
What are some important things I am missing to make this production worthy?
I believe I have to set the # of procs to ensure this uses maximum goroutines?
Should I be using output buffering?
Anything else you see missing that is best-practise?
var (
templates = template.Must(template.ParseFiles("templates/home.html")
)
func main() {
r := mux.NewRouter()
r.HandleFunc("/", WelcomeHandler)
http.ListenAndServe(":9000", r)
}
func WelcomeHandler(w http.ResponseWriter, r *http.Request) {
homePage, err := api.LoadHomePage()
if err != nil {
}
tmpl := "home"
renderTemplate(w, tmpl, homePage)
}
func renderTemplate(w http.ResponseWriter, tmpl string, hp *HomePage) {
err := templates.ExecuteTemplate(w, tmpl+".html", hp)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
You don't need to set/change runtime.GOMAXPROCS() as since Go 1.5 it defaults to the number of available CPU cores.
Buffering output? From the performance point of view, you don't need to. But there may be other considerations for which you may.
For example, your renderTemplate() function may potentially panic. If executing the template starts writing to the output, it involves setting the HTTP response code and other headers prior to writing data. And if a template execution error occurs after that, it will return an error, and so your code attempts to send back an error response. At this point HTTP headers are already written, and this http.Error() function will try to set headers again => panic.
One way to avoid this is to first render the template into a buffer (e.g. bytes.Buffer), and if no error is returned by the template execution, then you can write the content of the buffer to the response writer. If error occurs, then of course you won't write the content of the buffer, but send back an error response just like you did.
To sum it up, your code is production ready performance-wise (excluding the way you handle template execution errors).
WelcomeHandler should return when err != nil is true.
Log the error when one is hit to help investigation.
Place templates = template.Must(template.ParseFiles("templates/home.html") in the init. Split it into separate lines. If template.ParseFiles returns an then error make a Fatal log. And if you have multiple templates to initialize then initialize them in goroutines with a common WaitGroup to speed up the startup.
Since you are using mux, HTTP Server is too clean with its URLs might also be good to know.
You might also want to reconsider the decision of letting the user's know why they got the http.StatusInternalServerError response.
Setting the GOMAXPROCS > 1 if you have more the one core would definitely be a good idea but I would keep it less than number of cores available.
I searched around and as far as I can tell, POST form requests are already limited to 10MB (http://golang.org/src/net/http/request.go#L721).
If I were to go about reducing this in my ServeHTTP method, I'm not sure how to properly do it. I would try something like this:
r.Body = http.MaxBytesReader(w, r.Body, MaxFileSize)
err := r.ParseForm()
if err != nil {
//redirect to some error page
return
}
But would returning upon error close the connection as well? How would I prevent having to read everything? I found this: https://stackoverflow.com/a/26393261/2202497, but what if content length is not set and in the middle of reading I realize that the file is too big.
I'm using this as a security measure to prevent someone from hogging my server's resources.
The correct way to limit the size of the request body is to do as you suggested:
r.Body = http.MaxBytesReader(w, r.Body, MaxFileSize)
err := r.ParseForm()
if err != nil {
// redirect or set error status code.
return
}
MaxBytesReader sets a flag on the response when the limit is reached. When this flag is set, the server does not read the remainder of the request body and the server closes the connection on return from the handler.
If you are concerned about malicious clients, then you should also set Server.ReadTimeout, Server.WriteTimeout and possibly Server.MaxHeaderBytes.
If you want to set the request body limit for all of your handlers, then wrap root handler with a handler that sets the limit before delegating to the root handler:
type maxBytesHandler struct {
h http.Handler
n int64
}
func (h *maxBytesHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, h.n)
h.h.ServeHTTP(w, r)
}
Wrap the root handler when calling ListenAndServe:
log.Fatal(http.ListenAndServe(":8080", &maxBytesHandler{h:mux, n:4096))
or when configuring a server:
s := http.Server{
Addr: ":8080",
Handler: &maxBytesReader{h:mux, n:4096},
}
log.Fatal(s.ListenAndServe())
There's no need for a patch as suggested in another answer. MaxBytesReader is the official way to limit the size of the request body.
Edit: As others cited MaxByteReader is the supported way. It is interesting that the default reader is instead, limitreader after type asserting for max byte reader.
Submit a patch to the Go source code and make it configurable! You are working with an open source project after all. Adding a setter to http.Request and some unit tests for it is probably only 20 minutes worth of work. Having a hardcoded value here is a bit clunky, give back and fix it :).
You can of course implement your own ParseForm(r *http.Request) method if you really need to override this. Go is essentially BSD, so you can copy paste the library ParseForm and change the limit, but thats a bit ugly no?