How do you securely serve a file in golang - go

I'm new to developing web applications. I'm working with golang and would like to serve user uploaded files securely, such as allowing them to view their own files only.
Now I have saved those files to a local file system with random names. If I serve the entire directory, malicious users may view other users files. This sounds like a common use case, I wonder what's the best approach to deal with it?

This question in pretty vague and architectural decisions must be made to optimize data access and secure the files.
However, here is a simple solution that might serve your use-case.
package main
import (
"fmt"
"mime"
"net/http"
"path/filepath"
)
//UserFilesMap is the map that contains
var UserFilesMap map[string]FilePermission
type FilePermission map[string]struct{}
//FileServer is the function that serves files
func FileServer(w http.ResponseWriter, r *http.Request) {
//get the file path the user wants to access
filename := r.URL.Path[9:]
var uname, pass string
var ok bool
if uname, pass, ok = r.BasicAuth(); !ok {
w.WriteHeader(http.StatusForbidden)
return
}
if !(uname == "user" && pass == "1234") {
w.WriteHeader(http.StatusForbidden)
return
}
//Checking if user has permission to the file
if _, ok := UserFilesMap[uname][filename]; !ok {
w.WriteHeader(http.StatusForbidden)
return
}
w.Header().Set("Content-Type", mime.TypeByExtension(filepath.Ext(filename)))
http.ServeFile(w, r, "files/"+filename)
}
func main() {
UserFilesMap = make(map[string]FilePermission)
// UserFilesMap["user"] = FilePermission{"xyz.txt": struct{}{}}
UserFilesMap["user"] = FilePermission{"abc.txt": struct{}{}}
http.HandleFunc("/getFile/", FileServer)
if err := http.ListenAndServe(":8080", nil); err != nil {
fmt.Println("Error in ListenAndServe")
}
}
Here, I used a map to store the permissions of files. I would suggest you go for a SQL table instead.

If ur filenames are random and long enough and use a secure random generator this is already secure (unless directory-listing is enabled) with some limits though.
https://golang.org/pkg/crypto/rand/
One user will have access to the file only if he has the url with the random name. The limitation is though that the URL will be saved in the browser history, if someone else finds it he will also have access to it.

Related

How to use Go to get the Github commit history of a given file of a repository

Like the title said, my question here is how to use Go to programmatically get the Github commit history of a given file in a given repository
It seems that you need to access GitHub api from golang. There are a plenty of libraries but I would recommend using go-github.
Here is how you can try doing that
package main
import (
"context"
"github.com/google/go-github/github"
)
func main() {
var username string = "MayukhSobo"
client := github.NewClient(nil)
commits, response, err := client.Repositories.ListCommits(context.Background(), username, "Awesome-Snippets", nil)
if err != nil && response.StatusCode != 200 {
panic(err)
}
for _, commit := range commits {
// commit.SHA
// commit.Files
// You can use the commit
}
}
If you are trying to access the some other public repo, you need to pass the owner name in the username and change the repo name.
If you face access issues, it can be probably it is a private repo. You can also use the key pair to set up the access.

gorilla/mux golang cache static files

I have a go web app which serves static HTML/JS/CSS files plus has some API endpoints. I noticed that my HTML/JS/CSS are not being cached on the browser. E.g., every time I reload a page, they are fully re-downloaded.
Is this a server side configuration change that I need to set? How can I accomplish this with Go and Gorilla Mux?
I'm using Google App Engine, so Nginx is not a possibility.
Here's my main.go code:
package main
import (
"associations"
"html/template"
"net/http"
"log"
"io/ioutil"
"github.com/gorilla/mux"
"github.com/rs/cors"
"google.golang.org/appengine"
"google.golang.org/appengine/mail"
)
var index = template.Must(template.ParseFiles(
"dist/index.html",
))
func init() {
r := mux.NewRouter()
r.HandleFunc("/", homeHandler)
r.HandleFunc("/api/{tenant}/certificates", associations.GetCertificate).Methods("GET")
r.HandleFunc("/api/{tenant}/requests", associations.PostRequest).Methods("POST")
// handle static files
r.PathPrefix("/static/").Handler(
http.StripPrefix("/static/", http.FileServer(http.Dir("dist/static/"))))
r.NotFoundHandler = http.HandlerFunc(homeHandler) // work around for SPA serving index.html
handler := cors.Default().Handler(r)
http.Handle("/", handler)
}
EDIT: Here's the solution with #Topo's suggestion:
// handle static files
r.PathPrefix("/static/").Handler(http.StripPrefix("/static/",
CacheControlWrapper(http.FileServer(http.Dir("dist/static/")))))
....
func CacheControlWrapper(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
h.ServeHTTP(w, r)
})
}
To tell the browser to cache your files, you need to tell it for how long, if not the user would never see newer versions of the file.
To do this just set the Cache-Control header inside your handler function:
w.Header().Set("Cache-Control", "max-age=3600")
It's also a good idea to use the Etag header to let the browsers know when there is a new version of the file. That way you can cache the files for a really long time, and still serve the new content to the users as soon as it's available:
etag := "some unique value"
w.Header().Set("Etag", etag)
You need to use a different etag value every time the file changes. The browser stores it and only reloads the file when the etag is different to the stored value. You can use something like the file name + modified date for the etag:
var modTime time.Time
fi, err := fh.Stat()
if err == nil {
modTime = fi.ModTime()
} else {
modTime = time.Now()
}
etag := "\"" + file + modTime.String() + "\""
w.Header().Set("Etag", etag)
You can read the mozilla docs for the Cache-Control and Etag heders.
If you want to avoid writing your own handler function for static files and instead keep using http.FileServer, you will need to wrap it in a handler that sets the headers before writing the response. Here is a blog post on wrapping web handlers.

How to make templates work with gin framework?

I am newbie to golang.
To learn it I have started with a simple web app using gin framework.
I have followed the gin doc & configured template file but not able to make it work. I am getting an error -
panic: html/template: pattern matches no files: `templates/*`
goroutine 1 [running]:
html/template.Must
/usr/local/Cellar/go/1.5.2/libexec/src/html/template/template.go:330
github.com/gin-gonic/gin.(*Engine).LoadHTMLGlob
/Users/ameypatil/deployment/go/src/github.com/gin-gonic/gin/gin.go:126
main.main()
/Users/ameypatil/deployment/go/src/github.com/ameykpatil/gospike/main.go:17
Below is my code -
package main
import (
"net/http"
"github.com/gin-gonic/gin"
)
func main() {
//os.Setenv("GIN_MODE", "release")
//gin.SetMode(gin.ReleaseMode)
// Creates a gin router with default middleware:
// logger and recovery (crash-free) middleware
router := gin.Default()
router.LoadHTMLGlob("templates/*")
//router.LoadHTMLFiles("templates/index.tmpl")
router.GET("/", func(c *gin.Context) {
c.HTML(http.StatusOK, "index.tmpl", gin.H{
"title": "GoSpike",
})
})
// By default it serves on :8080 unless a
// PORT environment variable was defined.
router.Run(":4848")
}
My directory structure is
- gospike
--- templates
------index.tmpl
--- main.go
go install command does not give any error
but on actually running, it gives the above error. I searched & there were similar issues logged on gin's github repo but they are closed now.
I have tried various things but I guess I am missing something obvious. What am I missing?
I'm guessing the issue is that you're using a relative filepath to access your templates.
If I compile and run your code from the gospike directory, it works fine. But if I run gospike from any other directory, I get the same error you were seeing.
So either you need to always run gospike in the parent directory of templates, or you need to use the absolute path. You could either hard code it:
router.LoadHTMLGlob("/go/src/github.com/ameykpatil/gospike/templates/*")
or you could do something like
router.LoadHTMLGlob(filepath.Join(os.Getenv("GOPATH"),
"src/github.com/ameykpatil/gospike/templates/*"))
but that will fail if you have multiple paths set in your GOPATH. A better long-term solution might be setting a special environment variable like TMPL_DIR, and then just using that:
router.LoadHTMLGlob(filepath.Join(os.Getenv("TMPL_DIR"), "*"))
use relative path glob will working, you can try to code
router.LoadHTMLGlob("./templates/*")
notice the . dot sign which meaning current directory, gin.Engine will load template
base on templates subdirectory of current directory .
Here is how I do it. This walks through the directory and collects the files marked with my template suffix which is .html & then I just include all of those. I haven't seen this answer anywhere so I thought Id post it.
// START UP THE ROUTER
router := gin.Default()
var files []string
filepath.Walk("./views", func(path string, info os.FileInfo, err error) error {
if strings.HasSuffix(path, ".html") {
files = append(files, path)
}
return nil
})
router.LoadHTMLFiles(files...)
// SERVE STATICS
router.Use(static.Serve("/css", static.LocalFile("./css", true)))
router.Use(static.Serve("/js", static.LocalFile("./js", true)))
router.Use(static.Serve("/images", static.LocalFile("./images", true)))
routers.LoadBaseRoutes(router)
routers.LoadBlog(router)
router.Run(":8080")
There is a multitemplate HTML render to support multi tempaltes.
You can use AddFromFiles to add multi files:
r.AddFromFiles("index", "templates/base.html", "templates/index.html")
r.AddFromFiles("article", "templates/base.html", "templates/index.html", "templates/article.html")
Or load multi files under a directory:
package main
import (
"path/filepath"
"github.com/gin-contrib/multitemplate"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.New()
r.HTMLRender = loadTemplates()
// ...
}
func loadTemplates() multitemplate.Render {
files, err := filepath.Glob("template/*.tmpl")
if err != nil {
panic("failed to load html templates: " + err.Error())
}
r := multitemplate.New()
// Generate our templates map from our template/ directories
for _, file := range files {
r.AddFromFiles(filepath.Base(file), file)
}
// add other html templates directly
r.Add("test.tmpl", someTemplate)
return r
}
You could see more APIs in this repo you want, hope this post help :)

Google Cloud Bigtable authentication with Go

I'm trying to insert a simple record as in GoDoc. But this returns,
rpc error: code = 7 desc = "User can't access project: tidy-groove"
When I searched for grpc codes, it says..
PermissionDenied Code = 7
// Unauthenticated indicates the request does not have valid
// authentication credentials for the operation.
I've enabled Big table in my console and created a cluster and a service account and recieved the json. What I'm doing wrong here?
package main
import (
"fmt"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/cloud"
"google.golang.org/cloud/bigtable"
"io/ioutil"
)
func main() {
fmt.Println("Start!")
put()
}
func getClient() *bigtable.Client {
jsonKey, err := ioutil.ReadFile("TestProject-7854ea9op741.json")
if err != nil {
fmt.Println(err.Error())
}
config, err := google.JWTConfigFromJSON(
jsonKey,
bigtable.Scope,
) // or bigtable.AdminScope, etc.
if err != nil {
fmt.Println(err.Error())
}
ctx := context.Background()
client, err := bigtable.NewClient(ctx, "tidy-groove", "asia-east1-b", "test1-bigtable", cloud.WithTokenSource(config.TokenSource(ctx)))
if err != nil {
fmt.Println(err.Error())
}
return client
}
func put() {
ctx := context.Background()
client := getClient()
tbl := client.Open("table1")
mut := bigtable.NewMutation()
mut.Set("links", "maps.google.com", bigtable.Now(), []byte("1"))
mut.Set("links", "golang.org", bigtable.Now(), []byte("1"))
err := tbl.Apply(ctx, "com.google.cloud", mut)
if err != nil {
fmt.Println(err.Error())
}
}
I've solved the problem. It's nothing wrong with the code, but config json itself. So anyone who out there want to authenticate and came here by google search... This code is correct and working perfectly. What I've done wrong is follows.
First I made a service account and got the json. But google warned me that im not an owner of project hence it wont be added to accept list but anyway it let me download the json.
Then I deleted that key from console and requested project owner to create a key for me.
There he has created another key with the same name I given.. And since he's the owner no error/warning msgs displayed and successfully json file was downloaded.
When I tried with that... my question begun. That's when i posted this question.
After that with no solutions. I asked owner to delete that key and create another key but with a different name..
Then it worked! It seems if you try to create a key with non-owner account and then again create with same name ( after deleting original of course ) has no effect. Hope this helps everyone out there :)
Take a look at: helloworld.go or search.go which uses GOOGLE_APPLICATION_CREDENTIALS environment variable.
For most environments, you no longer even need to set GOOGLE_APPLICATION_CREDENTIALS. Google Cloud Platform, Managed VMs or Google App Engine all have the right thing set for you. Your desktop environment will also be correct if you've used gcloud init or it's predecessor gcloud auth login followed by gcloud config set project <projectID>.

Read template in init or in handler function?

I'm writing a basic server for a website. Now I face a (for me) difficult performance question. Is it better to read the template file in the init() function?
// Initialize all pages of website
func init(){
indexPageData, err := ioutil.ReadFile("./tpl/index.tpl")
check(err)
}
Or in the http.HandlerFunc?
func index(w http.ResponseWriter, req *http.Request){
indexPageData, err := ioutil.ReadFile("./tpl/index.tpl")
check(err)
indexPageTpl := template.Must(template.New("index").Parse(string(indexPageData)))
indexPageTpl.Execute(w, "test")
}
I think in the first example, after the server is started you have no need to access the disk and increase the performance of the request.
But during development I want to refresh the browser and see the new content. That can be done with the second example.
Does someone have a state-of-the-art solution? Or what is the right from the performance point of view?
Let's analyze the performance:
We name your first solution (with slight changes, see below) a and your second solution b.
One request:
a: One disk access
b: One disk access
Ten requests:
a: One disk access
b: Ten disk accesses
10 000 000 requests:
a: One disk access
b: 10 000 000 disk accesses (this is slow)
So, performance is better with your first solution. But what about your concern regarding up-to-date data? From the documentation of func (t *Template) Execute(wr io.Writer, data interface{}) error:
Execute applies a parsed template to the specified data object, writing the output to wr. If an error occurs executing the template or writing its output, execution stops, but partial results may already have been written to the output writer. A template may be executed safely in parallel.
So, what happens is this:
You read a template from disk
You parse the file into a template
You choose the data to fill in the blanks with
You Execute the template with that data, the result is written out into an io.Writer
Your data is as up-to-date as you choose it. This has nothing to do with re-reading the template from disk, or even re-parsing it. This is the whole idea behind templates: One disk access, one parse, multiple dynamic end results.
The documentation quoted above tells us another thing:
A template may be executed safely in parallel.
This is very useful, because your http.HandlerFuncs are ran in parallel, if you have multiple requests in parallel.
So, what to do now?
Read the template file once,
Parse the template once,
Execute the template for every request.
I'm not sure if you should read and parse in the init() function, because at least the Must can panic (and don't use some relative, hard coded path in there!) - I would try to do that in a more controlled environment, e.g. provide a function (like New()) to create a new instance of your server and do that stuff in there.
EDIT: I re-read your question and I might have misunderstood you:
If the template itself is still in development then yes, you would have to read it on every request to have an up-to-date result. This is more convenient than to restart the server every time you change the template. For production, the template should be fixed and only the data should change.
Sorry if I got you wrong there.
Never read and parse template files in the request handler in production, that is as bad as it can get (you should like always avoid this). During development it is ok of course.
Read this question for more details:
It takes too much time when using "template" package to generate a dynamic web page to client in golang
You could approach this in multiple ways. Here I list 4 with example implementation.
1. With a "dev mode" setting
You could have a constant or variable telling if you're running in development mode which means templates are not to be cached.
Here's an example to that:
const dev = true
var indexTmpl *template.Template
func init() {
if !dev { // Prod mode, read and cache template
indexTmpl = template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
}
}
func getIndexTmpl() *template.Template {
if dev { // Dev mode, always read fresh template
return template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
} else { // Prod mode, return cached template
return indexTmpl
}
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
getIndexTmpl().Execute(w, "test")
}
2. Specify in the request (as a param) if you want a fresh template
When you develop, you may specify an extra URL parameter indicating to read a fresh template and not use the cached one, e.g. http://localhost:8080/index?dev=true
Example implementation:
var indexTmpl *template.Template
func init() {
indexTmpl = getIndexTmpl()
}
func getIndexTmpl() *template.Template {
return template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
t := indexTmpl
if r.FormValue("dev") != nil {
t = getIndexTmpl()
}
t.Execute(w, "test")
}
3. Decide based on host
You can also check the host name of the request URL, and if it is "localhost", you can omit the cache and use a fresh template. This requires the smallest extra code and effort. Note that you may want to accept other hosts as well e.g. "127.0.0.1" (up to you what you want to include).
Example implementation:
var indexTmpl *template.Template
func init() {
indexTmpl = getIndexTmpl()
}
func getIndexTmpl() *template.Template {
return template.Must(template.New("index").ParseFiles(".tpl/index.tpl"))
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
t := indexTmpl
if r.URL.Host == "localhost" || strings.HasPrefix(r.URL.Host, "localhost:") {
t = getIndexTmpl()
}
t.Execute(w, "test")
}
4. Check template file last modified
You could also store the last modified time of the template file when it is loaded. Whenever the template is requested, you can check the last modified time of the source template file. If it has changed, you can reload it before executing it.
Example implementation:
type mytempl struct {
t *template.Template
lastmod time.Time
mutex sync.Mutex
}
var indexTmpl mytempl
func init() {
// You may want to call this in init so first request won't be slow
checkIndexTempl()
}
func checkIndexTempl() {
nm := ".tpl/index.tpl"
fi, err := os.Stat(nm)
if err != nil {
panic(err)
}
if indexTmpl.lastmod != fi.ModTime() {
// Changed, reload. Don't forget the locking!
indexTmpl.mutex.Lock()
defer indexTmpl.mutex.Unlock()
indexTmpl.t = template.Must(template.New("index").ParseFiles(nm))
indexTmpl.lastmod = fi.ModTime()
}
}
func indexHandler(w http.ResponseWriter, r *http.Request) {
checkIndexTempl()
indexTmpl.t.Execute(w, "test")
}

Resources