I have a go web app which serves static HTML/JS/CSS files plus has some API endpoints. I noticed that my HTML/JS/CSS are not being cached on the browser. E.g., every time I reload a page, they are fully re-downloaded.
Is this a server side configuration change that I need to set? How can I accomplish this with Go and Gorilla Mux?
I'm using Google App Engine, so Nginx is not a possibility.
Here's my main.go code:
package main
import (
"associations"
"html/template"
"net/http"
"log"
"io/ioutil"
"github.com/gorilla/mux"
"github.com/rs/cors"
"google.golang.org/appengine"
"google.golang.org/appengine/mail"
)
var index = template.Must(template.ParseFiles(
"dist/index.html",
))
func init() {
r := mux.NewRouter()
r.HandleFunc("/", homeHandler)
r.HandleFunc("/api/{tenant}/certificates", associations.GetCertificate).Methods("GET")
r.HandleFunc("/api/{tenant}/requests", associations.PostRequest).Methods("POST")
// handle static files
r.PathPrefix("/static/").Handler(
http.StripPrefix("/static/", http.FileServer(http.Dir("dist/static/"))))
r.NotFoundHandler = http.HandlerFunc(homeHandler) // work around for SPA serving index.html
handler := cors.Default().Handler(r)
http.Handle("/", handler)
}
EDIT: Here's the solution with #Topo's suggestion:
// handle static files
r.PathPrefix("/static/").Handler(http.StripPrefix("/static/",
CacheControlWrapper(http.FileServer(http.Dir("dist/static/")))))
....
func CacheControlWrapper(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
h.ServeHTTP(w, r)
})
}
To tell the browser to cache your files, you need to tell it for how long, if not the user would never see newer versions of the file.
To do this just set the Cache-Control header inside your handler function:
w.Header().Set("Cache-Control", "max-age=3600")
It's also a good idea to use the Etag header to let the browsers know when there is a new version of the file. That way you can cache the files for a really long time, and still serve the new content to the users as soon as it's available:
etag := "some unique value"
w.Header().Set("Etag", etag)
You need to use a different etag value every time the file changes. The browser stores it and only reloads the file when the etag is different to the stored value. You can use something like the file name + modified date for the etag:
var modTime time.Time
fi, err := fh.Stat()
if err == nil {
modTime = fi.ModTime()
} else {
modTime = time.Now()
}
etag := "\"" + file + modTime.String() + "\""
w.Header().Set("Etag", etag)
You can read the mozilla docs for the Cache-Control and Etag heders.
If you want to avoid writing your own handler function for static files and instead keep using http.FileServer, you will need to wrap it in a handler that sets the headers before writing the response. Here is a blog post on wrapping web handlers.
Related
I'm new to developing web applications. I'm working with golang and would like to serve user uploaded files securely, such as allowing them to view their own files only.
Now I have saved those files to a local file system with random names. If I serve the entire directory, malicious users may view other users files. This sounds like a common use case, I wonder what's the best approach to deal with it?
This question in pretty vague and architectural decisions must be made to optimize data access and secure the files.
However, here is a simple solution that might serve your use-case.
package main
import (
"fmt"
"mime"
"net/http"
"path/filepath"
)
//UserFilesMap is the map that contains
var UserFilesMap map[string]FilePermission
type FilePermission map[string]struct{}
//FileServer is the function that serves files
func FileServer(w http.ResponseWriter, r *http.Request) {
//get the file path the user wants to access
filename := r.URL.Path[9:]
var uname, pass string
var ok bool
if uname, pass, ok = r.BasicAuth(); !ok {
w.WriteHeader(http.StatusForbidden)
return
}
if !(uname == "user" && pass == "1234") {
w.WriteHeader(http.StatusForbidden)
return
}
//Checking if user has permission to the file
if _, ok := UserFilesMap[uname][filename]; !ok {
w.WriteHeader(http.StatusForbidden)
return
}
w.Header().Set("Content-Type", mime.TypeByExtension(filepath.Ext(filename)))
http.ServeFile(w, r, "files/"+filename)
}
func main() {
UserFilesMap = make(map[string]FilePermission)
// UserFilesMap["user"] = FilePermission{"xyz.txt": struct{}{}}
UserFilesMap["user"] = FilePermission{"abc.txt": struct{}{}}
http.HandleFunc("/getFile/", FileServer)
if err := http.ListenAndServe(":8080", nil); err != nil {
fmt.Println("Error in ListenAndServe")
}
}
Here, I used a map to store the permissions of files. I would suggest you go for a SQL table instead.
If ur filenames are random and long enough and use a secure random generator this is already secure (unless directory-listing is enabled) with some limits though.
https://golang.org/pkg/crypto/rand/
One user will have access to the file only if he has the url with the random name. The limitation is though that the URL will be saved in the browser history, if someone else finds it he will also have access to it.
I'm writing a basic server for a website. Now I face a (for me) difficult performance question. Is it better to read the template file in the init() function?
// Initialize all pages of website
func init(){
indexPageData, err := ioutil.ReadFile("./tpl/index.tpl")
check(err)
}
Or in the http.HandlerFunc?
func index(w http.ResponseWriter, req *http.Request){
indexPageData, err := ioutil.ReadFile("./tpl/index.tpl")
check(err)
indexPageTpl := template.Must(template.New("index").Parse(string(indexPageData)))
indexPageTpl.Execute(w, "test")
}
I think in the first example, after the server is started you have no need to access the disk and increase the performance of the request.
But during development I want to refresh the browser and see the new content. That can be done with the second example.
Does someone have a state-of-the-art solution? Or what is the right from the performance point of view?
Let's analyze the performance:
We name your first solution (with slight changes, see below) a and your second solution b.
One request:
a: One disk access
b: One disk access
Ten requests:
a: One disk access
b: Ten disk accesses
10 000 000 requests:
a: One disk access
b: 10 000 000 disk accesses (this is slow)
So, performance is better with your first solution. But what about your concern regarding up-to-date data? From the documentation of func (t *Template) Execute(wr io.Writer, data interface{}) error:
Execute applies a parsed template to the specified data object, writing the output to wr. If an error occurs executing the template or writing its output, execution stops, but partial results may already have been written to the output writer. A template may be executed safely in parallel.
So, what happens is this:
You read a template from disk
You parse the file into a template
You choose the data to fill in the blanks with
You Execute the template with that data, the result is written out into an io.Writer
Your data is as up-to-date as you choose it. This has nothing to do with re-reading the template from disk, or even re-parsing it. This is the whole idea behind templates: One disk access, one parse, multiple dynamic end results.
The documentation quoted above tells us another thing:
A template may be executed safely in parallel.
This is very useful, because your http.HandlerFuncs are ran in parallel, if you have multiple requests in parallel.
So, what to do now?
Read the template file once,
Parse the template once,
Execute the template for every request.
I'm not sure if you should read and parse in the init() function, because at least the Must can panic (and don't use some relative, hard coded path in there!) - I would try to do that in a more controlled environment, e.g. provide a function (like New()) to create a new instance of your server and do that stuff in there.
EDIT: I re-read your question and I might have misunderstood you:
If the template itself is still in development then yes, you would have to read it on every request to have an up-to-date result. This is more convenient than to restart the server every time you change the template. For production, the template should be fixed and only the data should change.
Sorry if I got you wrong there.
Never read and parse template files in the request handler in production, that is as bad as it can get (you should like always avoid this). During development it is ok of course.
Read this question for more details:
It takes too much time when using "template" package to generate a dynamic web page to client in golang
You could approach this in multiple ways. Here I list 4 with example implementation.
1. With a "dev mode" setting
You could have a constant or variable telling if you're running in development mode which means templates are not to be cached.
Here's an example to that:
const dev = true
var indexTmpl *template.Template
func init() {
if !dev { // Prod mode, read and cache template
indexTmpl = template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
}
}
func getIndexTmpl() *template.Template {
if dev { // Dev mode, always read fresh template
return template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
} else { // Prod mode, return cached template
return indexTmpl
}
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
getIndexTmpl().Execute(w, "test")
}
2. Specify in the request (as a param) if you want a fresh template
When you develop, you may specify an extra URL parameter indicating to read a fresh template and not use the cached one, e.g. http://localhost:8080/index?dev=true
Example implementation:
var indexTmpl *template.Template
func init() {
indexTmpl = getIndexTmpl()
}
func getIndexTmpl() *template.Template {
return template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
t := indexTmpl
if r.FormValue("dev") != nil {
t = getIndexTmpl()
}
t.Execute(w, "test")
}
3. Decide based on host
You can also check the host name of the request URL, and if it is "localhost", you can omit the cache and use a fresh template. This requires the smallest extra code and effort. Note that you may want to accept other hosts as well e.g. "127.0.0.1" (up to you what you want to include).
Example implementation:
var indexTmpl *template.Template
func init() {
indexTmpl = getIndexTmpl()
}
func getIndexTmpl() *template.Template {
return template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
t := indexTmpl
if r.URL.Host == "localhost" || strings.HasPrefix(r.URL.Host, "localhost:") {
t = getIndexTmpl()
}
t.Execute(w, "test")
}
4. Check template file last modified
You could also store the last modified time of the template file when it is loaded. Whenever the template is requested, you can check the last modified time of the source template file. If it has changed, you can reload it before executing it.
Example implementation:
type mytempl struct {
t *template.Template
lastmod time.Time
mutex sync.Mutex
}
var indexTmpl mytempl
func init() {
// You may want to call this in init so first request won't be slow
checkIndexTempl()
}
func checkIndexTempl() {
nm := ".tpl/index.tpl"
fi, err := os.Stat(nm)
if err != nil {
panic(err)
}
if indexTmpl.lastmod != fi.ModTime() {
// Changed, reload. Don't forget the locking!
indexTmpl.mutex.Lock()
defer indexTmpl.mutex.Unlock()
indexTmpl.t = template.Must(template.New("index").ParseFiles(nm))
indexTmpl.lastmod = fi.ModTime()
}
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
checkIndexTempl()
indexTmpl.t.Execute(w, "test")
}
I am building a webpage. The page should be able to handle the different http methods (GET, POST...). My page technically works and handles each type of request, but in the case of GET requests (serving index.html at the root "/" path), my page is not displaying properly. None of the images or css are displaying correctly, presumably because those files cannot be found.
I have noticed that http.Handle provides better results than http.HandleFunc when substituted into my server.go code below, in that the images and css do display correctly using:
http.FileServer(http.Dir("static"))
http.Handle("/", http.StripPrefix("/", fs))
The following is my web server with images and css not being displayed correctly. In general, my intent is to use static files for everything, including html (ex. index.html) and to implement some solution using nothing but standard go.
server.go code
package main
import (
"net/http"
"fmt"
)
func indexHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Add("Content Type", "text/html")
switch r.Method {
case "GET":
http.ServeFile(w, r, "./static/index.html")
case "POST":
fmt.Pprintf(w, "POST!")
case "PUT":
fmt.Pprintf(w, "PUT!")
case "DELETE":
fmt.Pprintf(w, "DELETE!")
default:
fmt.Pprintf(w, "Default!")
}
}
func main() {
http.HandleFunc("/", indexHandler)
http.ListenAndServe(":3000", nil)
}
You've hard-coded your server to always return index.html for any GET request, no matter what is being requested. So if your index.html includes a reference to style.css, the browser is going to make a second request for style.css and you'll return index.html again.
I assume what you're trying to do is have all GET requests return static files, while other verbs will do something else. You just need to pass those along to the file server:
root := "static"
...
case "GET":
if r.URL.Path == "" || r.URL.Path == "/" {
http.ServeFile(w, r, path.Join(root, "index.html"))
} else {
http.ServeFile(w, r, path.Join(root, r.URL.Path))
}
Note that by the time your handler is called, all ".." references in the URL have been removed, attackers can't use this to escape your static tree. But ServeFile() will return directory listings, so you'll need to check for that if that's a problem.
I'm a little bit confused. Much of examples shows usage of both: http.ServeFile(..) and http.FileServer(..), but seems they have very close functionality. Also I have found no information about how to set custom NotFound handler.
// This works and strip "/static/" fragment from path
fs := http.FileServer(http.Dir("static"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
// This works too, but "/static2/" fragment remains and need to be striped manually
http.HandleFunc("/static2/", func(w http.ResponseWriter, r *http.Request) {
http.ServeFile(w, r, r.URL.Path[1:])
})
http.ListenAndServe(":8080", nil)
I've tried to read source code and both of them use serveFile(ResponseWriter, *Request, FileSystem, string, bool) underlying function. However http.FileServer return fileHandler with its own ServeHTTP() method and make some preparation work before serving file (eg path.Clean()).
So why need this separation? Which method better to use? And how can I set custom NotFound handler, for example when requested file not found?
The main difference is that http.FileServer does effectively almost 1:1 mapping of an HTTP prefix with a filesystem. In plain english, it serves up an entire directory path. and all its children.
Say you had a directory called /home/bob/static and you had this setup:
fs := http.FileServer(http.Dir("/home/bob/static"))
http.Handle("/static/", http.StripPrefix("/static", fs))
Your server would take requests for e.g. /static/foo/bar and serve whatever is at /home/bob/static/foo/bar (or 404)
In contrast, the ServeFile is a lower level helper that can be used to implement something similar to FileServer, or implement your own path munging potentially, and any number of things. It simply takes the named local file and sends it over the HTTP connection. By itself, it won't serve a whole directory prefix (unless you wrote a handler that did some lookup similar to FileServer)
NOTE Serving up a filesystem naively is a potentially dangerous thing (there are potentially ways to break out of the rooted tree) hence I recommend that unless you really know what you're doing, use http.FileServer and http.Dir as they include checks to make sure people can't break out of the FS, which ServeFile doesn't.
Addendum
Your secondary question, how do you do a custom NotFound handler, unfortunately, is not easily answered. Because this is called from internal function serveFile as you noticed, there's no super easy place to break into that. There are potentially some sneaky things like intercepting the response with your own ResponseWriter which intercepts the 404 response code, but I'll leave that exercise to you.
Here a handler which sends a redirect to "/" if file is not found. This comes in handy when adding a fallback for an Angular application, as suggested here, which is served from within a golang service.
Note: This code is not production ready. Only illustrative (at best :-)
package main
import "net/http"
type (
// FallbackResponseWriter wraps an http.Requesthandler and surpresses
// a 404 status code. In such case a given local file will be served.
FallbackResponseWriter struct {
WrappedResponseWriter http.ResponseWriter
FileNotFound bool
}
)
// Header returns the header of the wrapped response writer
func (frw *FallbackResponseWriter) Header() http.Header {
return frw.WrappedResponseWriter.Header()
}
// Write sends bytes to wrapped response writer, in case of FileNotFound
// It surpresses further writes (concealing the fact though)
func (frw *FallbackResponseWriter) Write(b []byte) (int, error) {
if frw.FileNotFound {
return len(b), nil
}
return frw.WrappedResponseWriter.Write(b)
}
// WriteHeader sends statusCode to wrapped response writer
func (frw *FallbackResponseWriter) WriteHeader(statusCode int) {
Log.Printf("INFO: WriteHeader called with code %d\n", statusCode)
if statusCode == http.StatusNotFound {
Log.Printf("INFO: Setting FileNotFound flag\n")
frw.FileNotFound = true
return
}
frw.WrappedResponseWriter.WriteHeader(statusCode)
}
// AddFallbackHandler wraps the handler func in another handler func covering authentication
func AddFallbackHandler(handler http.HandlerFunc, filename string) http.HandlerFunc {
Log.Printf("INFO: Creating fallback handler")
return func(w http.ResponseWriter, r *http.Request) {
Log.Printf("INFO: Wrapping response writer in fallback response writer")
frw := FallbackResponseWriter{
WrappedResponseWriter: w,
FileNotFound: false,
}
handler(&frw, r)
if frw.FileNotFound {
Log.Printf("INFO: Serving fallback")
http.Redirect(w, r, "/", http.StatusSeeOther)
}
}
}
It can be added as in this example (using goji as mux):
mux.Handle(pat.Get("/*"),
AddFallbackHandler(http.FileServer(http.Dir("./html")).ServeHTTP, "/"))
I'm trying to chain HTTP handlers in go to provide some added functionality, like this:
package router
import (
// snip
"github.com/gorilla/mux"
"github.com/gorilla/handlers"
"net/http"
)
// snip
r := mux.NewRouter()
/* routing code */
var h http.Handler
h = r
if useGzip {
h = handlers.CompressHandler(h)
}
if useLogFile {
fn := pathToLog
accessLog, err := os.OpenFile(fn, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0666)
if err != nil {
panic(err)
}
h = handlers.CombinedLoggingHandler(accessLog, h)
}
// etc...
The problem is, if any HTTP headers are already set by one of the controllers that the gorilla/mux router points to (for example, w.WriteHeader(404) or w.Header().Set("Content-Type", "application/json")) - this silently breaks any "wrapper" handler trying to set or add its own headers, like the compress handler. I can't see any errors, unless I forgot to catch one somewhere, but the browser gets an invalid response.
Is there any graceful way to deal with this, short of just stashing the headers somewhere and then leaving the final handler to write them? It seems like that would mean rewriting the handlers' code, which I'd love to avoid if at all possible.
Once you call w.WriteHeader(404), the header goes on a wire. So you can't add to it anymore.
Best way you can do is to buffer status code and write it at the end of a chain.
For example, you can provide your own wrapper for http.ResponseWriter that would re-implement WriteHeader() to save status value. Then add method Commit() to actually write it.
Call Commit() in the last handler. You have to determine somehow which handler is last, of course.
I experienced the same silently-failing behaviour. But only in handlers where I did WritheHeader to set a status code other than StatusOK. I think things went wrong in this part of CompressHandler:
if h.Get("Content-Type") == "" {
h.Set("Content-Type", http.DetectContentType(b))
}
Which appears to be resolved when explicitly setting the content type in my own handler:
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(code)