Go webserver - don't cache files using timestamp - caching

I'm running a webserver written in go on an embedded system. The timestamp of index.html may go backwards if someone has downgraded the firmware version. If index.html is older than the previous version, the server sends a http 304 response (not modified), and serves a cached version of the file.
The webserver code is using http.FileServer() and http.ListenAndServe().
The problem can easily reproduced by modifying the timestamp of index.html using the Posix command touch
touch -d"23:59" index.html
reloading the page, then
touch -d"23:58" index.html
reloading this time will give a 304 response on index.html.
Is there a way to prevent timestamp based caching?

Assuming your file server code is like the example in the docs:
http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("/static"))))
You can write a handler that sets the appropriate cache headers to prevent this behaviour by stripping ETag headers and setting Cache-Control: no-cache, private, max-age=0 to prevent caching (both locally and in upstream proxies):
var epoch = time.Unix(0, 0).Format(time.RFC1123)
var noCacheHeaders = map[string]string{
"Expires": epoch,
"Cache-Control": "no-cache, private, max-age=0",
"Pragma": "no-cache",
"X-Accel-Expires": "0",
}
var etagHeaders = []string{
"ETag",
"If-Modified-Since",
"If-Match",
"If-None-Match",
"If-Range",
"If-Unmodified-Since",
}
func NoCache(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
// Delete any ETag headers that may have been set
for _, v := range etagHeaders {
if r.Header.Get(v) != "" {
r.Header.Del(v)
}
}
// Set our NoCache headers
for k, v := range noCacheHeaders {
w.Header().Set(k, v)
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
Use it like so:
http.Handle("/static/", NoCache(http.StripPrefix("/static/", http.FileServer(http.Dir("/static")))))
Note: I originally wrote this at github.com/zenazn/goji/middleware, so you can also just import that, but it's a simple piece of code to write and I wanted to show a full example for posterity!

Related

Go Mime package (1.14) behaves differently locally compared to the official Docker image

I have upgraded my local Go version from 1.13 to 1.14 and then I updated a project I was working on via re-initializing using go mod.
Locally:
$ go version
go version go1.14 linux/amd64
go.mod of my project:
module example-project
go 1.14
There has been an update in the mime package in Go 1.14 that changes the default type of .js files from application/javascript to text/javascript.
I have an application that serves a folder with a JavaScript file in it like:
func main() {
http.HandleFunc("/static/", StaticHandler)
http.ListenAndServe(":3000", nil)
}
func StaticHandler(w http.ResponseWriter, r *http.Request) {
fs := http.StripPrefix("/static", http.FileServer(http.Dir("public/")))
fs.ServeHTTP(w, r)
}
I updated a test case to reflect the mime changes in Go 1.14:
func TestStaticHandlerServeJS(t *testing.T) {
req, err := http.NewRequest("GET", "/static/index.js", nil)
if err != nil {
t.Fatal(err)
}
rr := httptest.NewRecorder()
handler := http.HandlerFunc(StaticHandler)
handler.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
expected := "text/javascript; charset=utf-8"
if rr.Header().Get("Content-Type") != expected {
t.Errorf("handler returned unexpected Content-Type: got %v want %v",
rr.Header().Get("Content-Type"), expected)
}
}
When I run this locally, the test case which checks the Content-Type fails:
TestStaticHandlerServeJS: main_test.go:27: handler returned unexpected Content-Type: got application/javascript want text/javascript; charset=utf-8
I can also confirm in the browser that the file is indeed served with the Mime Type "application/javascript" as it was back in Go 1.13.
When I run this test on a Docker container using the official golang:1.14.0-alpine3.11 image, this test passes, and it reflects the changed behavior of the mime package.
So as a result, I'm left with a test case that fails locally and passes on the container. I maintain only a single version of Go locally, and that's 1.14 as I've shown above. What could be the reason why my local Go installation has the mime package behaving differently?
It was interesting to me too and I have had the same behaviour like you - go 1.14 delivered on my mashine (macOs catalina) application/javascript instead of text/javascript. I debugged the program and found this function in type.go of mime package:
func initMime() {
if fn := testInitMime; fn != nil {
fn()
} else {
setMimeTypes(builtinTypesLower, builtinTypesLower)
osInitMime()
}
}
interesting stuff is going on in else block. After setting builtInTypes where extention js is assigned to text/javascript there is os specific assignment of file extention to content type, which overwrites the built-in assignment. On mac it is going to file type_unix.go where files
"/etc/mime.types",
"/etc/apache2/mime.types",
"/etc/apache/mime.types",
are tested to be available and in my case a file /etc/apache2/mime.types was present in the os and it contains ... surprise a line
application/javascript js
and this line overwrites the go built-in definition for .js extention and results in Content-Type: application/javascript to be delivered to client and cause your test to fail.

Go httputil.ReverseProxy not overriding the Host header

I'm basically trying to write a reverse proxy server so that when I curl localhost:8080/get it proxies the request to https://nghttp2.org/httpbin/get.
Note: the https://nghttp2.org/httpbin/get service listed above is http/2. But this behavior happens with http/1 as well, such as https://httpbin.org/get.
I'm using httputil.ReverseProxy for this and I'm rewriting the URL while customizing the Host header to not to leak the localhost:8080 to the actual backend.
However, the request still hits the backend with Host: localhost:8080 no matter how many times I set it on the header. Similarly, I used mitmproxy to snoop on the request and it looks like the net/http.Client sets the :authority pseudo-header to localhost:8080
Here's my source code:
package main
import (
"log"
"net/http"
"net/http/httputil"
)
func main() {
proxy := &httputil.ReverseProxy{
Transport: roundTripper(rt),
Director: func(req *http.Request) {
req.URL.Scheme = "https"
req.URL.Host = "nghttp2.org"
req.URL.Path = "/httpbin" + req.URL.Path
req.Header.Set("Host", "nghttp2.org") // <--- I set it here first
},
}
log.Fatal(http.ListenAndServe(":8080", proxy))
}
func rt(req *http.Request) (*http.Response, error) {
log.Printf("request received. url=%s", req.URL)
req.Header.Set("Host", "nghttp2.org") // <--- I set it here as well
defer log.Printf("request complete. url=%s", req.URL)
return http.DefaultTransport.RoundTrip(req)
}
// roundTripper makes func signature a http.RoundTripper
type roundTripper func(*http.Request) (*http.Response, error)
func (f roundTripper) RoundTrip(req *http.Request) (*http.Response, error) { return f(req) }
When I query curl localhost:8080/get the request gets proxied to https://nghttp2.org/httpbin/get. The echoed response shows that clearly my directives setting the Host header didn't do anything:
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "localhost:8080",
"User-Agent": "curl/7.54.0"
},
"origin": "2601:602:9c02:16c2:fca3:aaab:3914:4a71",
"url": "https://localhost:8080/httpbin/get"
}
mitmproxy snooping also clearly shows that the request was made with :authority pseudo-header set to localhost:8080:
From http.Request docs:
// For server requests, Host specifies the host on which the URL
// is sought. Per RFC 7230, section 5.4, this is either the value
// of the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
// To prevent DNS rebinding attacks, server Handlers should
// validate that the Host header has a value for which the
// Handler considers itself authoritative. The included
// ServeMux supports patterns registered to particular host
// names and thus protects its registered Handlers.
//
// For client requests, Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string
So the value of URL.Host is only used in case request.Host is empty which is not the case. Setting request.Host should resolve the issue:
req.Host = "nghttp2.org"
Related issue discussed here.

Not able to get exact effective,final URL

Using the Golang's http.Get() i am able to get effective or final url after multiple redirects,but in few cases where there are 303 redirects and special characters in URL golang is being weird, i am not able to get the actual final url. Following is the example i am dealing with - "http://swiggy.com//google.com/%2f.." , if we open this url in browser we get redirected to google, but i couldnt get the same using http.Get()
Other side behavior may depends on a lot of factors - for example of your User-Agent used, Cookies, IP and so on. Also sometimes it can change because of DDoS protection mechanism or things alike.
You may modify your app and see how it passes redirect stages:
Result:
getURL: http://swiggy.com//google.com/%2f..
Redirecting: 301 https://swiggy.com/google.com/%2f..
Redirecting: 301 https://www.swiggy.com/google.com/%2f..
Redirecting: 303 https://www.swiggy.com/google.com/%2f../
finalURL: https://www.swiggy.com/google.com/%2f../
Req Headers: map[Referer:[https://www.swiggy.com/google.com/%2f..]]
Resp Headers: map[Date:[Mon, 06 Nov 2017 12:51:20 GMT] Content-Type:[text/html; charset=utf-8] Content-Security-Policy-Report-Only:[default-src 'self';script-src https://chuknu.sokrati.com/15946/ https://www.google-analytics.com/ https://cdn.inspectlet.com/ https://tracking.sokrati.com/ https://connect.facebook.net/ https://bam.nr-data.net/ https://maps.googleapis.com/ https://js-agent.newrelic.com/ https://www.googletagmanager.com/ https://s3-ap-southeast-1.amazonaws.com/static.swiggy/ https://*.juspay.in https://connect.facebook.net/ https://www.googletagmanager.com/ *.swiggy.in *.swiggy.com https://chat2.hotline.io/ 'self' 'unsafe-inline' 'unsafe-eval' 'nonce-150997268072300';style-src https://fonts.googleapis.com/ https://www.swiggy.com/ https://s3-ap-southeast-1.amazonaws.com/static.swiggy/ https://chat2.hotline.io/ 'self' 'unsafe-inline' 'unsafe-eval';img-src https://res.cloudinary.com/swiggy/ https://www.google-analytics.com/ https://www.google.co.in/ https://www.facebook.com/ https://tracking.sokrati.com/ http://api.swiggy.in/ https://api.swiggy.com https://d3oxf4lkkqx2kx.cloudfront.net/ https://maps.googleapis.com/ https://maps.gstatic.com/ https://csi.gstatic.com/ https://fonts.gstatic.com/ https://stats.g.doubleclick.net/ https://googleads.g.doubleclick.net/ https://www.google.com/ data: 'self'; font-src https://www.swiggy.com/ https://fonts.gstatic.com/ data: 'self';connect-src https://hn.inspectlet.com/ https://www.swiggy.com/ https://www.facebook.com/tr/ https://*.juspay.in/txns https://sentry.swiggyapp.com/ 'self';frame-src https://www.facebook.com/tr/ https://chat2.hotline.io/ https://*.webpush.hotline.io 'self';report-uri /csp/log] Etag:[W/"6f97-"] Vary:[Accept-Encoding] X-Data-Origin:[dweb_cluster/port-dweb-06 naxsi/waf rate-limiter-plain/rate-limiter-plain] X-Xss-Protection:[1; mode=block] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Frame-Options:[Deny] Set-Cookie:[__SW=sjfsljfd; Path=/]]
Modified code:
package main
import (
"fmt"
"net/http"
)
func CheckRedirect(r *http.Request, via []*http.Request) error {
fmt.Println("Redirecting:", r.Response.StatusCode, r.URL)
return nil
}
func main() {
getURL := "http://swiggy.com//google.com/%2f.."
fmt.Println("getURL:", getURL)
client := &http.Client{
CheckRedirect: CheckRedirect,
}
resp, err := client.Get(getURL)
if err != nil {
fmt.Println(err)
return
}
finalURL := resp.Request.URL.String()
fmt.Println("finalURL:", finalURL)
fmt.Println("Req Headers:", resp.Request.Header)
fmt.Println("Resp Headers:", resp.Header)
}

golang static stop index.html redirection

package main
import (
"log"
"net/http"
)
func main() {
fs := http.FileServer(http.Dir("."))
http.Handle("/", fs)
log.Println("Listening...")
http.ListenAndServe(":3000", nil)
}
So I have a index.html file and want server to stop showing it.
The docs for FileServer state that:
As a special case, the returned file server redirects any request
ending in "/index.html" to the same path, without the final
"index.html".
So /index.html is redirected to /, /foo/bar/index.html is redirected to /foo/bar/.
To avoid this register an additional handler for the special case.
http.HandleFunc("/index.html", func(w http.ResponseWriter, r *http.Request) {
f, err := os.Open("index.html")
if err != nil {
// handle error
return
}
http.ServeContent(w, r, "index.html", time.Now(), f)
})
Please note I'm using ServeContent insead of ServeFile because ServeFile handles /index.html requests the same way as FileServer.
There's no redirection going on, the default file to render when requesting a directory is index.html. The directory listing is a fallback for when this file isn't found, so you can't get a directory listing without removing the index.html file.
If you want a directory listing, you'll have to write it out yourself, which you can then format and style however you choose. The basic structure is very simple if you want to write it directly, take the internal dirList function for example:
w.Header().Set("Content-Type", "text/html; charset=utf-8")
fmt.Fprintf(w, "<pre>\n")
for _, d := range dirs {
name := d.Name()
if d.IsDir() {
name += "/"
}
url := url.URL{Path: name}
fmt.Fprintf(w, "%s\n", url.String(), htmlReplacer.Replace(name))
}
fmt.Fprintf(w, "</pre>\n")
I think there are 3 ways:
Make a PR which allow accessing /index.html without redirect to golang's net/http/fs library. https://github.com/golang/go/issues/53870
Hack your code: replace /index.html with / to stop 301 redirect, e.g., https://github.com/ahuigo/go-lib/commit/1a99191d5c01cf9025136ce8ddb9668f123de05c#diff-67e8621fbb99281a50c089bae53d4874663d3d21ca5e90809ec207c070029351R44
Customize your own http fs handler instead of official tools.

How to Make My Web Server written in Golang to support HTTP/2 Server Push?

My Web Server is Coded in Golang and supports HTTPS. I wish to leverage HTTP/2 Server Push features in the Web Server. The following Link explains how to convert HTTP Server to Support HTTP/2 :-
https://www.ianlewis.org/en/http2-and-go
However, it is not clear how to implement the Server Push notifications in Golang.
- How should I add the Server Push functionality ?
- How do I control, or manage, the documents and files to be Pushed ?
Go 1.7 and older do not support HTTP/2 server push in the standard library. Support for server push will be added in the upcoming 1.8 release (see the release notes, expected release is February).
With Go 1.8 you can use the new http.Pusher interface, which is implemented by net/http's default ResponseWriter. Pushers Push method returns ErrNotSupported, if server push is not supported (HTTP/1) or not allowed (the client has disabled server push).
Example:
package main
import (
"io"
"log"
"net/http"
)
func main() {
http.HandleFunc("/pushed", func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "hello server push")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if pusher, ok := w.(http.Pusher); ok {
if err := pusher.Push("/pushed", nil); err != nil {
log.Println("push failed")
}
}
io.WriteString(w, "hello world")
})
http.ListenAndServeTLS(":443", "server.crt", "server.key", nil)
}
If you want to use server push with Go 1.7 or older use can use the golang.org/x/net/http2 and write the frames directly.
As mentioned in other answers, you can make use of Go 1.8 feature (cast the writer to http.Pusher and then use the Push method).
That comes with a caveat: you must be serving the HTTP2 traffic right from your server.
If you're behind a proxy like NGINX, this might not work. If you want to consider that scenario, you can make use of the Link header to advertise the URLs to be pushed.
// In the case of HTTP1.1 we make use of the `Link` header
// to indicate that the client (in our case, NGINX) should
// retrieve a certain URL.
//
// See more at https://www.w3.org/TR/preload/#server-push-http-2.
func handleIndex(w http.ResponseWriter, r *http.Request) {
var err error
if *http2 {
pusher, ok := w.(http.Pusher)
if ok {
must(pusher.Push("/image.svg", nil))
}
} else {
// This ends up taking the effect of a server push
// when interacting directly with NGINX.
w.Header().Add("Link",
"</image.svg>; rel=preload; as=image")
}
w.Header().Add("Content-Type", "text/html")
_, err = w.Write(assets.Index)
must(err)
}
ps.: I wrote more about this here https://ops.tips/blog/nginx-http2-server-push/ if you're interested.

Resources