I'm developing a Go program based on Gin web framework.
I want to serve pictures from the local file system.
func main() {
r := gin.Default()
r.Static("/page2", "resources/pictures")
r.GET("/test", func(context *gin.Context) {
name := context.Query("name")
context.JSON(http.StatusOK, gin.H{
"name": name,
})
})
r.GET("/page", func(context *gin.Context) {
name := context.Query("picname")
context.File("resources/pictures/" + name + ".jpg")
context.JSON(http.StatusOK, "That is "+name)
})
r.Run(":9090")
}
I find it strange that when I use /page2 to get the pictures (url http://localhost:9090/page2/xiamei.jpg), it works fine.
But when I use /page to get the pictures (url http://localhost:9090/page/xiamei.jpg) an error happens.
http: wrote more than the declared Content-Length
What is the internal reason, and what is the root cause of the difference between these two access methods?
In the /page handler, you call both Context.File, and Context.JSON.
The first call to Context.File already writes the file to the response and sets Content-Length header.
By calling Context.JSON again with more payload, you are increasing the actual length of the content without updating the Content-Length header; this results in the error you see.
r.Static doesn't have this issue because it serves one file and doesn't add more payload than it should.
In a Gin handler you should call only one renderer function. If you are serving a file, you can remove the last c.JSON altogether.
Related
To my shame, I have not been able to figure out how to get data from the frontend in Gin framework. In Django I get data So:
user=request.data.get('user')
print(user)
Everything is simple and understandable as day.
How should I do it in gin?
user := c.Query("user")
user := c.Param("user")
user := c.Params.ByName("user")
user := c.PostForm("user")
println(user)//emptiness....
Well, I'd say you should fetch some book/HOWTO on how HTTP work and spend some time with it because it appears you're trying to bash the problem at hand without actually understanding what happens between your browser and your backend service.
The real problem here is that there are more moving parts that you appear to be aware of, and the way to go depends on what your frontent does.
You did not tell us exactly how you're doing your request,
but from a solicted comment it appears, you're using that "axios" tingy.
If I managed to google that project correctly,
its README states:
By default, axios serializes JavaScript objects to JSON. To send data in the application/x-www-form-urlencoded format instead, you can use one of the following options.
This means two things:
Unless you somehow tweaked the axios' settings, when you did
axios.post, is supposedly performed an HTTP POST request
with its Content-Type field set to application/json
and its payload (or "body" if you prefer) being
a JSON serialization of that {user:this.user} JavaScript object.
It's therefore futile to attempt to parse the query string.
And it's futile to attempt to parse the request as an HTTP form — which it isn't.
Instead, you supposedly want to interpret the incoming request's body as being JSON-formatted. I have no idea as to how to do that in "go-gin", but in plain Go that would be something like
func (h *Handler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
defer req.Body.Close()
var user User
dec := json.NewDecoder(req.Body)
err := dec.Decode(&user)
if err != nil {
rw.Header().Set("Content-Type", "text/plain; charset=UTF-8")
rw.WriteHeader(http.StatusBadRequest)
fmt.Fprintln(rw, "Error parsing request body: ", err)
return
}
}
And ideally you'd first check that the content type of the incoming request was indeed application/json and reject it right away with http.StatusBadRequest if it isn't.
An example of a working code to do that is
// VerifyContentTypeIsJSON makes sure the HTTP header of a server
// http.Request contains the Content-Type field and it indicates
// the request payload is JSON.
// The implementation is based on RFC 7231 (section 3.1.1.5) and RFC 8259.
func VerifyContentTypeIsJSON(header http.Header) error {
var s string
if values := header["Content-Type"]; len(values) > 0 {
s = values[0]
} else {
return errors.New("missing Content-Type")
}
if s == "" {
return errors.New("empty Content-Type")
}
if i := strings.IndexByte(s, ';'); i != -1 {
s = strings.TrimSpace(s[:i])
}
if strings.ToLower(s) != "application/json" {
return fmt.Errorf("unknown Content-Type: %v, must be application/json", s)
}
return nil
}
Having this function, you'd have something like this
after defer req.Body.Close() and actually parsing it:
if err := VerifyContentTypeIsJSON(req.Header); err != nil {
rw.Header().Set("Content-Type", "text/plain; charset=utf-8")
rw.WriteHeader(http.StatusBadRequest)
fmt.Fprintln(rw, err)
return
}
(Note that "go-gin" might have something akin to this already built-in, so please check this.)
The User type should be some struct type matching the shape of the JSON object you intend to unmarshal from the request. Something like this:
type User struct {
User string `json:"user"`
}
None that in the two places my example returned an
error to the user it used content type of plain text
(in UTF-8 encoding). This may be OK but may be not.
Say, your clients might expect a JSON-formatted document
of some agreed-upon shape.
Or you may use content negotiation, but I'd recommend to get simple things straight first.
Literature to check:
HTTP POST request explained at MDN.
URL's query string.
XHR explained at MDN — see also links there.
"Writing Web Applications in Go",
and this in general.
And to maybe answer that part of your question regarding
why it "just worked" in Django.
I can only guess, but I think it merely implements tons of magic which looks at the incoming request and tries to guess how to extract data from it.
The problem is that guessing may indeed work well for
one-off throwaway scripts, but when you're about implementing something like web API (what many not quite correctly call "REST", but let's not digress) it's best
to be very explicit about what your endpoint accept
precisely and how precisely they react to requests — both legitimate and non-well-formed.
Regarding magic in Go, you may read this.
you need to call c.Request.ParseForm() before using it from request.Form
says here:
For all requests, ParseForm parses the raw query from the URL and updates r.Form
For other HTTP methods, or when the Content-Type is not application/x-www-form-urlencoded, the request Body is not read, and r.PostForm is initialized to a non-nil, empty value.
If you're expecting a JSON body in the request, you can do it this way with gin. Create a struct for the data you want to grab out of the body. Use json tags for the JSON key names unless you're going to exactly match your Go field names to them. Then call the BindJSON method on the gin context.
For example:
type User struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
Login string `json:"login"`
}
func (h *Handler) HandleUser(gctx *gin.Context) {
user := new(User)
err := gctx.BindJSON(user)
if err != nil {
text := fmt.Sprintf("Failed to read user data from request: %s", err)
log.Error(text)
gctx.JSON(http.StatusBadRequest, gin.H{"error": text})
return
}
// do something with user
}
Server GIN can't handle routine default application/json requests from axios!!! What???
Requests should be sent as application/x-www-form-urlencoded.
My decision in Vue project:
Use vue-resource instead axios (axios.post=>this.$http.post) with option
Vue.http.options.emulateJSON = true; in main.js
So I'm using a go server to serve up a single page web application.
This works for serving all the assets on the root route. All the CSS and HTML are served up correctly.
fs := http.FileServer(http.Dir("build"))
http.Handle("/", fs)
So when the URL is http://myserverurl/index.html or http://myserverurl/styles.css, it serves the corresponding file.
But for a URL like http://myserverurl/myCustompage, it throws 404 if myCustompage is not a file in the build folder.
How do I make all routes for which a file does not exist serve index.html?
It is a single page web application and it will render the appropriate screen once the html and js are served. But it needs index.html to be served on routes for which there is no file.
How can this be done?
The handler returned by http.FileServer() does not support customization, it does not support providing a custom 404 page or action.
What we may do is wrap the handler returned by http.FileServer(), and in our handler we may do whatever we want of course. In our wrapper handler we will call the file server handler, and if that would send a 404 not found response, we won't send it to the client but replace it with a redirect response.
To achieve that, in our wrapper we create a wrapper http.ResponseWriter which we will pass to the handler returned by http.FileServer(), and in this wrapper response writer we may inspect the status code, and if it's 404, we may act to not send the response to the client, but instead send a redirect to /index.html.
This is an example how this wrapper http.ResponseWriter may look like:
type NotFoundRedirectRespWr struct {
http.ResponseWriter // We embed http.ResponseWriter
status int
}
func (w *NotFoundRedirectRespWr) WriteHeader(status int) {
w.status = status // Store the status for our own use
if status != http.StatusNotFound {
w.ResponseWriter.WriteHeader(status)
}
}
func (w *NotFoundRedirectRespWr) Write(p []byte) (int, error) {
if w.status != http.StatusNotFound {
return w.ResponseWriter.Write(p)
}
return len(p), nil // Lie that we successfully written it
}
And wrapping the handler returned by http.FileServer() may look like this:
func wrapHandler(h http.Handler) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
nfrw := &NotFoundRedirectRespWr{ResponseWriter: w}
h.ServeHTTP(nfrw, r)
if nfrw.status == 404 {
log.Printf("Redirecting %s to index.html.", r.RequestURI)
http.Redirect(w, r, "/index.html", http.StatusFound)
}
}
}
Note that I used http.StatusFound redirect status code instead of http.StatusMovedPermanently as the latter may be cached by browsers, and so if a file with that name is created later, the browser would not request it but display index.html immediately.
And now put this in use, the main() function:
func main() {
fs := wrapHandler(http.FileServer(http.Dir(".")))
http.HandleFunc("/", fs)
panic(http.ListenAndServe(":8080", nil))
}
Attempting to query a non-existing file, we'll see this in the log:
2017/11/14 14:10:21 Redirecting /a.txt3 to /index.html.
2017/11/14 14:10:21 Redirecting /favicon.ico to /index.html.
Note that our custom handler (being well-behaviour) also redirected the request to /favico.ico to index.html because I do not have a favico.ico file in my file system. You may want to add this as an exception if you don't have it either.
The full example is available on the Go Playground. You can't run it there, save it to your local Go workspace and run it locally.
Also check this related question: Log 404 on http.FileServer
Say I have a html/template like the following:
<html>
<body>
<p>{{SomeFunc .SomeData}}</p>
</body>
and sometimes SomeFunc returns an error. Is there an idiomatic way to deal with this?
If I write directly to the ResponseWriter, then a status code 200 has already been written before I encounter the error.
var tmpl *template.Template
func Handler(w http.ResponseWriter, r *http.Request) {
err := tmpl.Execute(w, data)
// "<html><body><p>" has already been written...
// what to do with err?
}
Preferably I would return a status code 400 or some such, but I can't see a way to do this if I use template.Execute directly on the ResponseWriter. Is there something I'm missing?
Since the template engine generates the output on-the-fly, parts of the template preceding the SomeFunc call are already sent to the output. And if the output is not buffered, they (along with the HTTP 200 status) may already be sent.
You can't do anything about that.
What you can do is perform the check before you call template.Execute(). In trivial case it should be enough to call SomeFunc() and check its return value. If you choose this path and the return value of SomeFunc() is complex, you do not have to call it again from the template, you can simply pass its return value to the params you pass to the template and refer to this value in the template (so SomeFunc() won't have to be executed twice).
If this is not enough or you can't control it, you can create a bytes.Buffer or strings.Builder, execute your template directed into this buffer, and after the Execute() returns, check if there were errors. If there were errors, send back a proper error message / page. If everything went ok, you can just send the content of the buffer to the ResponseWriter.
This could look something like this:
buf := &bytes.Buffer{}
err := tmpl.Execute(buf, data)
if err != nil {
// Send back error message, for example:
http.Error(w, "Hey, Request was bad!", http.StatusBadRequest) // HTTP 400 status
} else {
// No error, send the content, HTTP 200 response status implied
buf.WriteTo(w)
}
I'm a little bit confused. Much of examples shows usage of both: http.ServeFile(..) and http.FileServer(..), but seems they have very close functionality. Also I have found no information about how to set custom NotFound handler.
// This works and strip "/static/" fragment from path
fs := http.FileServer(http.Dir("static"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
// This works too, but "/static2/" fragment remains and need to be striped manually
http.HandleFunc("/static2/", func(w http.ResponseWriter, r *http.Request) {
http.ServeFile(w, r, r.URL.Path[1:])
})
http.ListenAndServe(":8080", nil)
I've tried to read source code and both of them use serveFile(ResponseWriter, *Request, FileSystem, string, bool) underlying function. However http.FileServer return fileHandler with its own ServeHTTP() method and make some preparation work before serving file (eg path.Clean()).
So why need this separation? Which method better to use? And how can I set custom NotFound handler, for example when requested file not found?
The main difference is that http.FileServer does effectively almost 1:1 mapping of an HTTP prefix with a filesystem. In plain english, it serves up an entire directory path. and all its children.
Say you had a directory called /home/bob/static and you had this setup:
fs := http.FileServer(http.Dir("/home/bob/static"))
http.Handle("/static/", http.StripPrefix("/static", fs))
Your server would take requests for e.g. /static/foo/bar and serve whatever is at /home/bob/static/foo/bar (or 404)
In contrast, the ServeFile is a lower level helper that can be used to implement something similar to FileServer, or implement your own path munging potentially, and any number of things. It simply takes the named local file and sends it over the HTTP connection. By itself, it won't serve a whole directory prefix (unless you wrote a handler that did some lookup similar to FileServer)
NOTE Serving up a filesystem naively is a potentially dangerous thing (there are potentially ways to break out of the rooted tree) hence I recommend that unless you really know what you're doing, use http.FileServer and http.Dir as they include checks to make sure people can't break out of the FS, which ServeFile doesn't.
Addendum
Your secondary question, how do you do a custom NotFound handler, unfortunately, is not easily answered. Because this is called from internal function serveFile as you noticed, there's no super easy place to break into that. There are potentially some sneaky things like intercepting the response with your own ResponseWriter which intercepts the 404 response code, but I'll leave that exercise to you.
Here a handler which sends a redirect to "/" if file is not found. This comes in handy when adding a fallback for an Angular application, as suggested here, which is served from within a golang service.
Note: This code is not production ready. Only illustrative (at best :-)
package main
import "net/http"
type (
// FallbackResponseWriter wraps an http.Requesthandler and surpresses
// a 404 status code. In such case a given local file will be served.
FallbackResponseWriter struct {
WrappedResponseWriter http.ResponseWriter
FileNotFound bool
}
)
// Header returns the header of the wrapped response writer
func (frw *FallbackResponseWriter) Header() http.Header {
return frw.WrappedResponseWriter.Header()
}
// Write sends bytes to wrapped response writer, in case of FileNotFound
// It surpresses further writes (concealing the fact though)
func (frw *FallbackResponseWriter) Write(b []byte) (int, error) {
if frw.FileNotFound {
return len(b), nil
}
return frw.WrappedResponseWriter.Write(b)
}
// WriteHeader sends statusCode to wrapped response writer
func (frw *FallbackResponseWriter) WriteHeader(statusCode int) {
Log.Printf("INFO: WriteHeader called with code %d\n", statusCode)
if statusCode == http.StatusNotFound {
Log.Printf("INFO: Setting FileNotFound flag\n")
frw.FileNotFound = true
return
}
frw.WrappedResponseWriter.WriteHeader(statusCode)
}
// AddFallbackHandler wraps the handler func in another handler func covering authentication
func AddFallbackHandler(handler http.HandlerFunc, filename string) http.HandlerFunc {
Log.Printf("INFO: Creating fallback handler")
return func(w http.ResponseWriter, r *http.Request) {
Log.Printf("INFO: Wrapping response writer in fallback response writer")
frw := FallbackResponseWriter{
WrappedResponseWriter: w,
FileNotFound: false,
}
handler(&frw, r)
if frw.FileNotFound {
Log.Printf("INFO: Serving fallback")
http.Redirect(w, r, "/", http.StatusSeeOther)
}
}
}
It can be added as in this example (using goji as mux):
mux.Handle(pat.Get("/*"),
AddFallbackHandler(http.FileServer(http.Dir("./html")).ServeHTTP, "/"))
I'm trying to chain HTTP handlers in go to provide some added functionality, like this:
package router
import (
// snip
"github.com/gorilla/mux"
"github.com/gorilla/handlers"
"net/http"
)
// snip
r := mux.NewRouter()
/* routing code */
var h http.Handler
h = r
if useGzip {
h = handlers.CompressHandler(h)
}
if useLogFile {
fn := pathToLog
accessLog, err := os.OpenFile(fn, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0666)
if err != nil {
panic(err)
}
h = handlers.CombinedLoggingHandler(accessLog, h)
}
// etc...
The problem is, if any HTTP headers are already set by one of the controllers that the gorilla/mux router points to (for example, w.WriteHeader(404) or w.Header().Set("Content-Type", "application/json")) - this silently breaks any "wrapper" handler trying to set or add its own headers, like the compress handler. I can't see any errors, unless I forgot to catch one somewhere, but the browser gets an invalid response.
Is there any graceful way to deal with this, short of just stashing the headers somewhere and then leaving the final handler to write them? It seems like that would mean rewriting the handlers' code, which I'd love to avoid if at all possible.
Once you call w.WriteHeader(404), the header goes on a wire. So you can't add to it anymore.
Best way you can do is to buffer status code and write it at the end of a chain.
For example, you can provide your own wrapper for http.ResponseWriter that would re-implement WriteHeader() to save status value. Then add method Commit() to actually write it.
Call Commit() in the last handler. You have to determine somehow which handler is last, of course.
I experienced the same silently-failing behaviour. But only in handlers where I did WritheHeader to set a status code other than StatusOK. I think things went wrong in this part of CompressHandler:
if h.Get("Content-Type") == "" {
h.Set("Content-Type", http.DetectContentType(b))
}
Which appears to be resolved when explicitly setting the content type in my own handler:
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(code)