I've been having some issues loading html templates using the Gin framework through the r.HTMLRender setting.
It would seem that the templates are not being found.
I have tried to use the following helpers:
GinHTMLRender: https://gist.github.com/madhums/4340cbeb36871e227905
EZ Gin Template: https://github.com/michelloworld/ez-gin-template
Neither of these seem to be working when setting the default path for templates (in my case app/views); for the purposes of getting to this to work my html template file structure looks like this:
/workspace
|-app
|-views
|-layouts
|-application.html
|-test.html
Here is a sample of the Gin loading code:
import (
"github.com/gin-contrib/location"
"github.com/gin-gonic/gin"
"fmt"
"os"
"github.com/gin-gonic/contrib/static"
"github.com/michelloworld/ez-gin-template"
)
//CORSMiddleware ...
func CORSMiddleware() gin.HandlerFunc {
/** CORS middleware **/
}
func Router() {
if os.Getenv("ENVIRONMENT") == "production" {
gin.SetMode(gin.ReleaseMode)
}
// Initialize Gin object
r := gin.Default()
// Cors Middleware
r.Use(CORSMiddleware())
// Rate limiting
rl, err := helpers.RateLimiterMiddleware()
if err != nil {
panic("Rate Limiting Initialization error")
}
r.Use(rl)
// Asset provision
r.Use(static.ServeRoot("/public","app/assets"))
// Get URL information
r.Use(location.Default())
// Attempt with EZ Template, fails
// I ge this error: "runtime error: invalid memory address or nil pointer dereference" when calling c.HTML(...)
render := eztemplate.New()
render.TemplatesDir = "app/views/" // default
render.Layout = "layouts/application" // default
render.Ext = ".html" // default
render.Debug = true // default
r.HTMLRender = render.Init()
// Attempt with GinHTMLRender, fails
// I get this error: https://gist.github.com/madhums/4340cbeb36871e227905#file-gin_html_render-go-L110
/*
htmlRender := GinHTMLRender.New()
htmlRender.TemplatesDir = "app/views/"
htmlRender.Debug = gin.IsDebugging()
htmlRender.Layout = "layouts/application"
log.Println("Dir:"+htmlRender.TemplatesDir)
r.HTMLRender = htmlRender.Create()*/
/** Some Routes **/
// Start web listener
r.Run(":8009") // listen and serve on 0.0.0.0:8080
}
The corresponding render call is code is the following:
/* c is of type *gin.Context */
c.HTML(200, "test", "")
For some reason it seems like the r.HTMLRender function is not taking into account the template path; I have attempted doing this:
_, err := template.ParseFiles("app/views/test.html")
if err != nil {
log.Println("Template Error")
} else {
log.Println("No Template Error")
}
This code consistently displays "No Template Error", which leads me to believe that the HTMLRender assignment is not considering the TemplatesDir set variable.
I've been stuck with this issue for some time, and I am not entirely sure how to get it resolved.
Any help getting this to work would be greatly appreciated.
After doing some further research I found the source of my problem with EZ Gin Template.
I'm hoping this helps anyone experiencing the same issue.
After taking a deeper look at the helper code, I realized that the template matching pattern is strict and does not search for files recursively; ie. it expects a specific file structure to find template files:
In the default setting, EZ Gin Template requires the following file structure to work:
/workspace
- app
- views
- layouts
- some_layout.html
- some_dir
- template_file.html
- _partial_template.html
- partials
- _some_other_partial.html
In order to allow for other file patterns, the helper set of functions needs to be modified.
In my case, I forked the helper code locally to allow matching 1st level template files:
func (r Render) Init() Render {
globalPartials := r.getGlobalPartials()
layout := r.TemplatesDir + r.Layout + r.Ext
// Match multiple levels of templates
viewDirs, _ := filepath.Glob(r.TemplatesDir + "**" + string(os.PathSeparator) + "*" + r.Ext)
// Added the following two lines to match for app/views/some_file.html as well as files on the **/*.html matching pattern
tmp, _ := filepath.Glob(r.TemplatesDir + "*" + r.Ext)
viewDirs = append(viewDirs, tmp...)
// Can be extended by replicating those two lines above and adding search paths within the base template path.
fullPartialDir := filepath.Join(r.TemplatesDir + r.PartialDir)
for _, view := range viewDirs {
templateFileName := filepath.Base(view)
//skip partials
if strings.Index(templateFileName, "_") != 0 && strings.Index(view, fullPartialDir) != 0 {
localPartials := r.findPartials(filepath.Dir(view))
renderName := r.getRenderName(view)
if r.Debug {
log.Printf("[GIN-debug] %-6s %-25s --> %s\n", "LOAD", view, renderName)
}
allFiles := []string{layout, view}
allFiles = append(allFiles, globalPartials...)
allFiles = append(allFiles, localPartials...)
r.AddFromFiles(renderName, allFiles...)
}
}
return r
}
I have not tried a similar solution with GinHTMLRenderer, but I expect that the issue might likely be related to it in terms of the expected file structure.
You can also bind the templates into the code. The jessevdk/go-assets-builder will generate a go file that contains assets within the specified directory. Make sure that the file generated is located where the main package is. Gin also provided this as an example in their Documentation For more Info. It will also include subfolders and its files (i. e. assets) in the binary.
Get The generator tool:
go get github.com/jessevdk/go-assets-builder
Generate:
# go-assets-builder <dir> -o <generated file name>
go-assets-builder app -o assets.go
Note that <generated file name> can also be like cmd/client/assets.go to specify the destination of the file to be generated.
Load Template:
package main
// ... imports
func main() {
r := gin.New()
t, err := loadTemplate()
if err != nil {
panic(err)
}
r.SetHTMLTemplate(t)
r.GET("/", func(c *gin.Context) {
c.HTML(http.StatusOK, "app/views/layouts/application.html", nil)
})
r.GET("/test", func(c *gin.Context) {
c.HTML(http.StatusOK, "app/views/test.html", nil)
})
r.Run(":8080")
}
// loadTemplate loads templates embedded by go-assets-builder
func loadTemplate() (*template.Template, error) {
t := template.New("")
// Assets is the templates
for name, file := range Assets.Files {
if file.IsDir() || !strings.HasSuffix(name, ".html") {
continue
}
h, err := ioutil.ReadAll(file)
if err != nil {
return nil, err
}
t, err = t.New(name).Parse(string(h))
if err != nil {
return nil, err
}
}
return t, nil
}
Here is how I do it. This walks through the directory and collects the files marked with my template suffix which is .html & then I just include all of those. I haven't seen this answer anywhere so I thought Id post it.
// START UP THE ROUTER
router := gin.Default()
var files []string
filepath.Walk("./views", func(path string, info os.FileInfo, err error) error {
if strings.HasSuffix(path, ".html") {
files = append(files, path)
}
return nil
})
router.LoadHTMLFiles(files...)
// SERVE STATICS
router.Use(static.Serve("/css", static.LocalFile("./css", true)))
router.Use(static.Serve("/js", static.LocalFile("./js", true)))
router.Use(static.Serve("/images", static.LocalFile("./images", true)))
routers.LoadBaseRoutes(router)
routers.LoadBlog(router)
router.Run(":8080")
Now they don't have to all be nested at the exact depth like the other suggestions ... the file structure can be uneven
Related
I am trying to build a string using text/template, where the template string could have arbitrary properties that are resolved via a map.
What I am trying to accomplish is identifying where one/any of the template properties is not resolved and return an error.
At the moment, I am using regexp but reaching out to the community of see if there was a better solution.
package main
import (
"bytes"
"fmt"
"regexp"
"text/template"
)
func main() {
data := "teststring/{{.someData}}/{{.notExist}}/{{.another}}"
// the issue here is that data can be arbitrary so i cannot do
// a lot of unknown if statements
t := template.Must(template.New("").Parse(data))
var b bytes.Buffer
fillers := map[string]interface{}{
"someData": "123",
"another": true,
// in this case, notExist is not defined, so the template will
// note resolve it
}
if err := t.Execute(&b, fillers); err != nil {
panic(err)
}
fmt.Println(b.String())
// teststring/123/<no value>/true
// here i am trying to catch if a the template required a value that was not provided
hasResolved := regexp.MustCompile(`<no value>`)
fmt.Println(hasResolved.MatchString(b.String()))
// add notExist to the fillers map
fillers["notExist"] = "testdata"
b.Reset()
if err := t.Execute(&b, fillers); err != nil {
panic(err)
}
fmt.Println(b.String())
fmt.Println(hasResolved.MatchString(b.String()))
// Output:
// teststring/123/<no value>/true
// true
// teststring/123/testdata/true
// false
}
You can let it fail by settings the options on the template:
func (t *Template) Option(opt ...string) *Template
"missingkey=default" or "missingkey=invalid"
The default behavior: Do nothing and continue execution.
If printed, the result of the index operation is the string
"<no value>".
"missingkey=zero"
The operation returns the zero value for the map type's element.
"missingkey=error"
Execution stops immediately with an error.
If you set it to missingkey=error, you get what what want.
t = t.Options("missingkey=error")
I want to create a yaml file from a current tmpl file. Basically I want to insert values in the sample.tmpl files stored in the /templates folder and create a new yaml file in the same folder sample.yml
My sample.tmpl looks like
url : {{ .host }}
namespace: {{ .namespace }}
I'm using the following function:
func ApplyTemplate(filePath string) (err error) {
// Variables - host, namespace
type Eingest struct {
host string
namespace string
}
ei := Eingest{host: "example.com", namespace: "finance"}
var templates *template.Template
var allFiles []string
files, err := ioutil.ReadDir(filePath)
if err != nil {
fmt.Println(err)
}
for _, file := range files {
filename := file.Name()
fullPath := filePath + "/" + filename
if strings.HasSuffix(filename, ".tmpl") {
allFiles = append(allFiles, fullPath)
}
}
fmt.Println("Files in path: ", allFiles)
// parses all .tmpl files in the 'templates' folder
templates, err = template.ParseFiles(allFiles...)
if err != nil {
fmt.Println(err)
}
s1 := templates.Lookup("sample.tmpl")
s1.ExecuteTemplate(os.Stdout, "sample.yml", ei)
fmt.Println()
return
}
s1.ExecuteTemplate() writes to stdout. How can I create a new file in the same folder? I believe something similar is used to build kubernetes yaml files. How do we achieve this using golang template package?
First: since you've already looked up the template, you should use template.Execute instead, but the same works with ExecuteTemplate.
text.Template.Execute takes a io.Writer as first argument. This is an interface with a single method: Write(p []byte) (n int, err error).
Any type that has that method implements the interface and can be used as a valid argument. One such type is a os.File. Simply create a new os.File object and pass it to Execute as follows:
// Build the path:
outputPath := filepath.Join(filepath, "sample.yml")
// Create the file:
f, err := os.Create(outputPath)
if err != nil {
panic(err)
}
defer f.Close() // don't forget to close the file when finished.
// Write template to file:
err = s1.Execute(f, ei)
if err != nil {
panic(err)
}
Note: don't forget to check whether s1 is nil, as documented in template.Lookup.
I want to specify an html template in one of my golang controller
My directory structure is like this
Project
-com
-src
- controller
-contoller.go
-view
- html
-first.html
I want to load first.html for request /new .I have used NewHandler for url /new and the NewHandler func is executing when /new request comes and is in controller.go. Here is my code
func NewHandler(w http.ResponseWriter, r *http.Request) {
t, err := template.ParseFiles("view/html/first.html")
if err == nil {
log.Println("Template parsed successfully....")
}
err := templates.ExecuteTemplate(w, "view/html/first.html", nil)
if err != nil {
log.Println("Not Found template")
}
// t.Execute(w, "")
}
But I am getting an error
panic: open first.html: no such file or directory
Please help me to remove this error. Thanks in advance
I have solved the issue by giving absolute path of the html. For that I created a class in which the html are parsed
package htmltemplates
import (
"html/template"
"path/filepath"
)
And in the NewHandler method I removed
//Templates is used to store all Templates
var Templates *template.Template
func init() {
filePrefix, _ := filepath.Abs("./work/src/Project/view/html/") // path from the working directory
Templates = template.Must(template.ParseFiles(filePrefix + "/first.html"))
...
//htmls must be specified here to parse it
}
And in the NewHandler I removed first 5 lines and instead gave
err := htmltemplates.Templates.ExecuteTemplate(w, "first.html", nil)
It is now working .But need a better solution if any
Have you included this line in the main function?
http.Handle("/view/", http.StripPrefix("/view/", http.FileServer(http.Dir("view"))))
view is the name of the directory that has to be specified in FileServer function to allow read/write.(view directory has to be kept in the same directory where your binary is present)
Better solution is to give absolute path name i.e instead of this "view/html/first.html",try this
func NewHandler(w http.ResponseWriter, r *http.Request) {
t, err := template.ParseFiles("./work/src/Project/view/html/first.html")
if err == nil {
log.Println("Template parsed successfully....")
}
err := templates.ExecuteTemplate(w, "view/html/first.html", nil)
if err != nil {
log.Println("Not Found template")
}
// t.Execute(w, "")
}
Solution, where I just traversed outside the directory without using libraries to find the path or convert them.
Using "os" package get the present working directory. Like,
import(
"os"
)
your_function(){
pwd,_ := os.pwd()}
Above snippet will return you the path where your program is trying to find the file.
Like
D:\Golang_Projects\POC\Distributed_Application_Go\cmd\teacherportal
After you're sure what path your program is looking at, use ".." to come outside of that directory. Like,
rootTemplate,err := template.ParseFiles("../../teacherportal/students.gohtml")
Alternative solutions are also there but I found this quite simple to implement.
I have the following working code to delete an object from Amazon s3
params := &s3.DeleteObjectInput{
Bucket: aws.String("Bucketname"),
Key : aws.String("ObjectKey"),
}
s3Conn.DeleteObjects(params)
But what i want to do is to delete all files under a folder using wildcard **. I know amazon s3 doesn't treat "x/y/file.jpg" as a folder y inside x but what i want to achieve is by mentioning "x/y*" delete all the subsequent objects having the same prefix. Tried amazon multi object delete
params := &s3.DeleteObjectsInput{
Bucket: aws.String("BucketName"),
Delete: &s3.Delete{
Objects: []*s3.ObjectIdentifier {
{
Key : aws.String("x/y/.*"),
},
},
},
}
result , err := s3Conn.DeleteObjects(params)
I know in php it can be done easily by s3->delete_all_objects as per this answer. Is the same action possible in GOlang.
Unfortunately the goamz package doesn't have a method similar to the PHP library's delete_all_objects.
However, the source code for the PHP delete_all_objects is available here (toggle source view): http://docs.aws.amazon.com/AWSSDKforPHP/latest/#m=AmazonS3/delete_all_objects
Here are the important lines of code:
public function delete_all_objects($bucket, $pcre = self::PCRE_ALL)
{
// Collect all matches
$list = $this->get_object_list($bucket, array('pcre' => $pcre));
// As long as we have at least one match...
if (count($list) > 0)
{
$objects = array();
foreach ($list as $object)
{
$objects[] = array('key' => $object);
}
$batch = new CFBatchRequest();
$batch->use_credentials($this->credentials);
foreach (array_chunk($objects, 1000) as $object_set)
{
$this->batch($batch)->delete_objects($bucket, array(
'objects' => $object_set
));
}
$responses = $this->batch($batch)->send();
As you can see, the PHP code will actually make an HTTP request on the bucket to first get all files matching PCRE_ALL, which is defined elsewhere as const PCRE_ALL = '/.*/i';.
You can only delete 1000 files at once, so delete_all_objects then creates a batch function to delete 1000 files at a time.
You have to create the same functionality in your go program as the goamz package doesn't support this yet. Luckily it should only be a few lines of code, and you have a guide from the PHP library.
It might be worth submitting a pull request for the goamz package once you're done!
Using the mc tool you can do:
mc rm -r --force https://BucketName.s3.amazonaws.com/x/y
it will delete all the objects with the prefix "x/y"
You can achieve the same with Go using minio-go like this:
package main
import (
"log"
"github.com/minio/minio-go"
)
func main() {
config := minio.Config{
AccessKeyID: "YOUR-ACCESS-KEY-HERE",
SecretAccessKey: "YOUR-PASSWORD-HERE",
Endpoint: "https://s3.amazonaws.com",
}
// find Your S3 endpoint here http://docs.aws.amazon.com/general/latest/gr/rande.html
s3Client, err := minio.New(config)
if err != nil {
log.Fatalln(err)
}
isRecursive := true
for object := range s3Client.ListObjects("BucketName", "x/y", isRecursive) {
if object.Err != nil {
log.Fatalln(object.Err)
}
err := s3Client.RemoveObject("BucketName", object.Key)
if err != nil {
log.Fatalln(err)
continue
}
log.Println("Removed : " + object.Key)
}
}
Since this question was asked, the AWS GoLang lib for S3 has received some new methods in S3 Manager to handle this task (in response to #Itachi's pr).
See Github record: https://github.com/aws/aws-sdk-go/issues/448#issuecomment-309078450
Here is their example in v1: https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/go/s3/DeleteObjects/DeleteObjects.go#L36
To get "wildcard matching" on paths inside the bucket, add the Prefix param to the example's ListObjectsInput call, as shown here:
iter := s3manager.NewDeleteListIterator(svc, &s3.ListObjectsInput{
Bucket: bucket,
Prefix: aws.String("somePathString"),
})
A bit late in the game, but since I was having the same problem, I created a small pkg that you can copy to your code base and import as needed.
func ListKeysInPrefix(s s3iface.S3API, bucket, prefix string) ([]string, error) {
res, err := s.Client.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String(prefix),
})
if err != nil {
return []string{}, err
}
var keys []string
for _, key := range res.Contents {
keys = append(keys, *key.Key)
}
return keys, nil
}
func createDeleteObjectsInput(keys []string) *s3.Delete {
rm := []*s3.ObjectIdentifier{}
for _, key := range keys {
rm = append(rm, &s3.ObjectIdentifier{Key: aws.String(key)})
}
return &s3.Delete{Objects: rm, Quiet: aws.Bool(false)}
}
func DeletePrefix(s s3iface.S3API, bucket, prefix string) error {
keys, err := s.ListKeysInPrefix(bucket, prefix)
if err != nil {
panic(err)
}
_, err = s.Client.DeleteObjects(&s3.DeleteObjectsInput{
Bucket: aws.String(bucket),
Delete: s.createDeleteObjectsInput(keys),
})
if err != nil {
return err
}
return nil
}
So, in the case you have a bucket called "somebucket" with the following structure: s3://somebucket/foo/some-prefixed-folder/bar/test.txt and wanted to delete from some-prefixed-folder onwards, usage would be:
func main() {
// create your s3 client here
// client := ....
err := DeletePrefix(client, "somebucket", "some-prefixed-folder")
if err != nil {
panic(err)
}
}
This implementation only allows to delete a maximum of 1000 entries from the given prefix due ListObjectsV2 implementation - but it is paginated, so it's a matter of adding the functionality to keep refreshing results until results are < 1000.
I'm looking for a way to invalidate cached static content upon version change. Preferably using commit id to invalidate. Is there anyway to do this in revel framework ?
I would prefer if its automatic but I could live with updating it each time if its a single place I have to edit.
The current strategy I have is changing the name of the static content route to include version but this requires several changes. In places that feel unnatural, for instance in the routing file.
You could do it manually via a config variable and an intercept method.
resourceversion.go
Create this file in your controllers folder:
package controllers
import (
"github.com/revel/revel"
)
// interceptor method, called before every request.
// Sets a template variable to the resourceVersion read from app.conf
func SetVersion(c *revel.Controller) revel.Result {
c.RenderArgs["resourceVersion"] = revel.Config.StringDefault("resourceVersion", "1")
return nil
}
init.go
In the init() method, append this line:
revel.InterceptMethod(controllers.SetVersion, revel.BEFORE)
templates
In your templates, where you want to use the resource version:
<link rel="stylesheet" type="text/css" href="/public/css/style.css?{{.resourceVersion}}">
app.conf
And finally, the place you will update it - add this line above the dev section to apply to dev and prod, or have a different one in each, whatever suits.
resourceVersion=20150716
I guess you could create a script as part of your build and release process that would automatically edit this config variable.
I did the things that Colin Nicholson suggested but I also created a controller called staticversionbasedcacheinvalidator and placed it in the controller folder. You can find it below. It allows you to ignore the first part of the request string allowing you to use a wildcard path to your public folder. For instance I use this route config row
GET /public/*filepath StaticVersionbasedCacheInvalidator.Serve("public")
I then added the {{.resourceVersion}} in my route instead of as a query param.
package controllers
import (
"github.com/revel/revel"
"os"
fpath "path/filepath"
"strings"
"syscall"
)
type StaticVersionbasedCacheInvalidator struct {
*revel.Controller
}
// This method handles requests for files. The supplied prefix may be absolute
// or relative. If the prefix is relative it is assumed to be relative to the
// application directory. The filepath may either be just a file or an
// additional filepath to search for the given file. This response may return
// the following responses in the event of an error or invalid request;
// 403(Forbidden): If the prefix filepath combination results in a directory.
// 404(Not found): If the prefix and filepath combination results in a non-existent file.
// 500(Internal Server Error): There are a few edge cases that would likely indicate some configuration error outside of revel.
//
// Note that when defining routes in routes/conf the parameters must not have
// spaces around the comma.
// Bad: StaticVersionbasedCacheInvalidator.Serve("public/img", "favicon.png")
// Good: StaticVersionbasedCacheInvalidator.Serve("public/img","favicon.png")
//
// Examples:
// Serving a directory
// Route (conf/routes):
// GET /public/{<.*>filepath} StaticVersionbasedCacheInvalidator.Serve("public")
// Request:
// public/js/sessvars.js
// Calls
// StaticVersionbasedCacheInvalidator.Serve("public","js/sessvars.js")
//
// Serving a file
// Route (conf/routes):
// GET /favicon.ico StaticVersionbasedCacheInvalidator.Serve("public/img","favicon.png")
// Request:
// favicon.ico
// Calls:
// StaticVersionbasedCacheInvalidator.Serve("public/img", "favicon.png")
func (c StaticVersionbasedCacheInvalidator) Serve(prefix, filepath string) revel.Result {
firstSplice := strings.Index(filepath,"/")
if(firstSplice != -1) {
filepath = filepath[firstSplice:len(filepath)];
}
// Fix for #503.
prefix = c.Params.Fixed.Get("prefix")
if prefix == "" {
return c.NotFound("")
}
return serve(c, prefix, filepath)
}
// This method allows modules to serve binary files. The parameters are the same
// as StaticVersionbasedCacheInvalidator.Serve with the additional module name pre-pended to the list of
// arguments.
func (c StaticVersionbasedCacheInvalidator) ServeModule(moduleName, prefix, filepath string) revel.Result {
// Fix for #503.
prefix = c.Params.Fixed.Get("prefix")
if prefix == "" {
return c.NotFound("")
}
var basePath string
for _, module := range revel.Modules {
if module.Name == moduleName {
basePath = module.Path
}
}
absPath := fpath.Join(basePath, fpath.FromSlash(prefix))
return serve(c, absPath, filepath)
}
// This method allows StaticVersionbasedCacheInvalidator serving of application files in a verified manner.
func serve(c StaticVersionbasedCacheInvalidator, prefix, filepath string) revel.Result {
var basePath string
if !fpath.IsAbs(prefix) {
basePath = revel.BasePath
}
basePathPrefix := fpath.Join(basePath, fpath.FromSlash(prefix))
fname := fpath.Join(basePathPrefix, fpath.FromSlash(filepath))
// Verify the request file path is within the application's scope of access
if !strings.HasPrefix(fname, basePathPrefix) {
revel.WARN.Printf("Attempted to read file outside of base path: %s", fname)
return c.NotFound("")
}
// Verify file path is accessible
finfo, err := os.Stat(fname)
if err != nil {
if os.IsNotExist(err) || err.(*os.PathError).Err == syscall.ENOTDIR {
revel.WARN.Printf("File not found (%s): %s ", fname, err)
return c.NotFound("File not found")
}
revel.ERROR.Printf("Error trying to get fileinfo for '%s': %s", fname, err)
return c.RenderError(err)
}
// Disallow directory listing
if finfo.Mode().IsDir() {
revel.WARN.Printf("Attempted directory listing of %s", fname)
return c.Forbidden("Directory listing not allowed")
}
// Open request file path
file, err := os.Open(fname)
if err != nil {
if os.IsNotExist(err) {
revel.WARN.Printf("File not found (%s): %s ", fname, err)
return c.NotFound("File not found")
}
revel.ERROR.Printf("Error opening '%s': %s", fname, err)
return c.RenderError(err)
}
return c.RenderFile(file, revel.Inline)
}