I'm looking for a way to invalidate cached static content upon version change. Preferably using commit id to invalidate. Is there anyway to do this in revel framework ?
I would prefer if its automatic but I could live with updating it each time if its a single place I have to edit.
The current strategy I have is changing the name of the static content route to include version but this requires several changes. In places that feel unnatural, for instance in the routing file.
You could do it manually via a config variable and an intercept method.
resourceversion.go
Create this file in your controllers folder:
package controllers
import (
"github.com/revel/revel"
)
// interceptor method, called before every request.
// Sets a template variable to the resourceVersion read from app.conf
func SetVersion(c *revel.Controller) revel.Result {
c.RenderArgs["resourceVersion"] = revel.Config.StringDefault("resourceVersion", "1")
return nil
}
init.go
In the init() method, append this line:
revel.InterceptMethod(controllers.SetVersion, revel.BEFORE)
templates
In your templates, where you want to use the resource version:
<link rel="stylesheet" type="text/css" href="/public/css/style.css?{{.resourceVersion}}">
app.conf
And finally, the place you will update it - add this line above the dev section to apply to dev and prod, or have a different one in each, whatever suits.
resourceVersion=20150716
I guess you could create a script as part of your build and release process that would automatically edit this config variable.
I did the things that Colin Nicholson suggested but I also created a controller called staticversionbasedcacheinvalidator and placed it in the controller folder. You can find it below. It allows you to ignore the first part of the request string allowing you to use a wildcard path to your public folder. For instance I use this route config row
GET /public/*filepath StaticVersionbasedCacheInvalidator.Serve("public")
I then added the {{.resourceVersion}} in my route instead of as a query param.
package controllers
import (
"github.com/revel/revel"
"os"
fpath "path/filepath"
"strings"
"syscall"
)
type StaticVersionbasedCacheInvalidator struct {
*revel.Controller
}
// This method handles requests for files. The supplied prefix may be absolute
// or relative. If the prefix is relative it is assumed to be relative to the
// application directory. The filepath may either be just a file or an
// additional filepath to search for the given file. This response may return
// the following responses in the event of an error or invalid request;
// 403(Forbidden): If the prefix filepath combination results in a directory.
// 404(Not found): If the prefix and filepath combination results in a non-existent file.
// 500(Internal Server Error): There are a few edge cases that would likely indicate some configuration error outside of revel.
//
// Note that when defining routes in routes/conf the parameters must not have
// spaces around the comma.
// Bad: StaticVersionbasedCacheInvalidator.Serve("public/img", "favicon.png")
// Good: StaticVersionbasedCacheInvalidator.Serve("public/img","favicon.png")
//
// Examples:
// Serving a directory
// Route (conf/routes):
// GET /public/{<.*>filepath} StaticVersionbasedCacheInvalidator.Serve("public")
// Request:
// public/js/sessvars.js
// Calls
// StaticVersionbasedCacheInvalidator.Serve("public","js/sessvars.js")
//
// Serving a file
// Route (conf/routes):
// GET /favicon.ico StaticVersionbasedCacheInvalidator.Serve("public/img","favicon.png")
// Request:
// favicon.ico
// Calls:
// StaticVersionbasedCacheInvalidator.Serve("public/img", "favicon.png")
func (c StaticVersionbasedCacheInvalidator) Serve(prefix, filepath string) revel.Result {
firstSplice := strings.Index(filepath,"/")
if(firstSplice != -1) {
filepath = filepath[firstSplice:len(filepath)];
}
// Fix for #503.
prefix = c.Params.Fixed.Get("prefix")
if prefix == "" {
return c.NotFound("")
}
return serve(c, prefix, filepath)
}
// This method allows modules to serve binary files. The parameters are the same
// as StaticVersionbasedCacheInvalidator.Serve with the additional module name pre-pended to the list of
// arguments.
func (c StaticVersionbasedCacheInvalidator) ServeModule(moduleName, prefix, filepath string) revel.Result {
// Fix for #503.
prefix = c.Params.Fixed.Get("prefix")
if prefix == "" {
return c.NotFound("")
}
var basePath string
for _, module := range revel.Modules {
if module.Name == moduleName {
basePath = module.Path
}
}
absPath := fpath.Join(basePath, fpath.FromSlash(prefix))
return serve(c, absPath, filepath)
}
// This method allows StaticVersionbasedCacheInvalidator serving of application files in a verified manner.
func serve(c StaticVersionbasedCacheInvalidator, prefix, filepath string) revel.Result {
var basePath string
if !fpath.IsAbs(prefix) {
basePath = revel.BasePath
}
basePathPrefix := fpath.Join(basePath, fpath.FromSlash(prefix))
fname := fpath.Join(basePathPrefix, fpath.FromSlash(filepath))
// Verify the request file path is within the application's scope of access
if !strings.HasPrefix(fname, basePathPrefix) {
revel.WARN.Printf("Attempted to read file outside of base path: %s", fname)
return c.NotFound("")
}
// Verify file path is accessible
finfo, err := os.Stat(fname)
if err != nil {
if os.IsNotExist(err) || err.(*os.PathError).Err == syscall.ENOTDIR {
revel.WARN.Printf("File not found (%s): %s ", fname, err)
return c.NotFound("File not found")
}
revel.ERROR.Printf("Error trying to get fileinfo for '%s': %s", fname, err)
return c.RenderError(err)
}
// Disallow directory listing
if finfo.Mode().IsDir() {
revel.WARN.Printf("Attempted directory listing of %s", fname)
return c.Forbidden("Directory listing not allowed")
}
// Open request file path
file, err := os.Open(fname)
if err != nil {
if os.IsNotExist(err) {
revel.WARN.Printf("File not found (%s): %s ", fname, err)
return c.NotFound("File not found")
}
revel.ERROR.Printf("Error opening '%s': %s", fname, err)
return c.RenderError(err)
}
return c.RenderFile(file, revel.Inline)
}
Related
I am working with lot of config files. I need to read all those individual config file in their own struct and then make one giant Config struct which holds all other individual config struct in it.
Let's suppose if I am working with 3 config files.
ClientConfig deals with one config file.
DataMapConfig deals with second config file.
ProcessDataConfig deals with third config file.
I created separate class for each of those individual config file and have separate Readxxxxx method in them to read their own individual config and return struct back. Below is my config.go file which is called via Init method from main function after passing path and logger.
config.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type Config struct {
ClientMapConfigs ClientConfig
DataMapConfigs DataMapConfig
ProcessDataConfigs ProcessDataConfig
}
func Init(path string, logger log.Logger) (*Config, error) {
var err error
clientConfig, err := ReadClientMapConfig(path, logger)
dataMapConfig, err := ReadDataMapConfig(path, logger)
processDataConfig, err := ReadProcessDataConfig(path, logger)
if err != nil {
return nil, err
}
return &Config{
ClientMapConfigs: *clientConfig,
DataMapConfigs: *dataMapConfig,
ProcessDataConfigs: *processDataConfig,
}, nil
}
clientconfig.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type ClientConfig struct {
.....
.....
}
const (
ClientConfigFile = "clientConfigMap.json"
)
func ReadClientMapConfig(path string, logger log.Logger) (*ClientConfig, error) {
files, err := utilities.FindFiles(path, ClientConfigFile)
// read all the files
// do some validation on all those files
// deserialize them into ClientConfig struct
// return clientconfig object back
}
datamapconfig.go
Similar style I have for datamapconfig too. Exactly replica of clientconfig.go file but operating on different config file name and will return DataMapConfig struct back.
processdataConfig.go
Same thing as clientconfig.go file. Only difference is it will operate on different config file and return ProcessDataConfig struct back.
Problem Statement
I am looking for ideas where this above design can be improved? Is there any better way to do this in golang? Can we use interface or anything else which can improve the above design?
If I have let's say 10 different files instead of 3, then do I need to keep doing above same thing for remaining 7 files? If yes, then the code will look ugly. Any suggestions or ideas will greatly help me.
Update
Everything looks good but I have few questions as I am confuse on how can I achieve those with your current suggestion. On majority of my configs, your suggestion is perfect but there are two cases on two different configs where I am confuse on how to do it.
Case 1 After deserializing json into original struct which matches json format, I make another different struct after massaging that data and then I return that struct back.
Case 2 All my configs have one file but there are few configs which have multiple files in them and the number isn't fixed. So I pass regex file name and then I find all the files starting with that regex and then loop over all those files one by one. After deserializing each json file, I start populating another object and keep populating it until all files have been deserialized and then make a new struct with those objects and then return it.
Example of above scenarios:
Sample case 1
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type CustomerManifest struct {
CustomerManifest map[int64]Site
}
type CustomerConfigs struct {
CustomerConfigurations []Site `json:"customerConfigurations"`
}
type Site struct {
....
....
}
const (
CustomerConfigFile = "abc.json"
)
func ReadCustomerConfig(path string, logger log.Logger) (*CustomerManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, CustomerConfigFile)
if err != nil {
return nil, err
}
var customerConfig CustomerConfigs
// there is only file for this config so loop will run once
for _, file := range files {
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &customerConfig)
if err != nil {
return nil, err
}
}
customerConfigIndex := BuildIndex(customerConfig, logger)
return &CustomerManifest{CustomerManifest: customerConfigIndex}, nil
}
func BuildIndex(customerConfig CustomerConfigs, logger log.Logger) map[int64]Site {
...
...
}
As you can see above in sample case 1, I am making CustomerManifest struct from CustomerConfigs struct and then return it instead of returning CustomerConfigs directly.
Sample case 2
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type StateManifest struct {
NotionTemplates NotionTemplates
NotionIndex map[int64]NotionTemplates
}
type NotionMapConfigs struct {
NotionTemplates []NotionTemplates `json:"notionTemplates"`
...
}
const (
// there are many files starting with "state-", it's not fixed number
StateConfigFile = "state-*.json"
)
func ReadStateConfig(path string, logger log.Logger) (*StateManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, StateConfigFile)
if err != nil {
return nil, err
}
var defaultTemp NotionTemplates
var idx = map[int64]NotionTemplates{}
// there are lot of config files for this config so loop will run multiple times
for _, file := range files {
var notionMapConfig NotionMapConfigs
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, ¬ionMapConfig)
if err != nil {
return nil, err
}
for _, tt := range notionMapConfig.NotionTemplates {
if tt.IsProcess {
defaultTemp = tt
} else if tt.ClientId > 0 {
idx[tt.ClientId] = tt
}
}
}
stateManifest := StateManifest{
NotionTemplates: defaultTemp,
NotionIndex: idx,
}
return &stateManifest, nil
}
As you can see above in my both the cases, I am making another different struct after deserializing is done and then I return that struct back but as of now in your current suggestion I think I won't be able to do this generically because for each config I do different type of massaging and then return those struct back. Is there any way to achieve above functionality with your current suggestion? Basically for each config if I want to do some massaging, then I should be able to do it and return new modified struct back but for some cases if I don't want to do any massaging then I can return direct deserialize json struct back. Can this be done generically?
Since there are config which has multiple files in them so that is why I was using my utilities.FindFiles method to give me all files basis on file name or regex name and then I loop over all those files to either return original struct back or return new struct back after massaging original struct data.
You can use a common function to load all the configuration files.
Assume you have config structures:
type Config1 struct {...}
type Config2 struct {...}
type Config3 struct {...}
You define configuration validators for those who need it:
func (c Config1) Validate() error {...}
func (c Config2) Validate() error {...}
Note that these implement a Validatable interface:
type Validatable interface {
Validate() error
}
There is one config type that includes all these configurations:
type Config struct {
C1 Config1
C2 Config2
C3 Config3
...
}
Then, you can define a simple configuration loader function:
func LoadConfig(fname string, out interface{}) error {
data, err:=ioutil.ReadFile(fname)
if err!=nil {
return err
}
if err:=json.Unmarshal(data,out); err!=nil {
return err
}
// Validate the config if necessary
if validator, ok:=out.(Validatable); ok {
if err:=validator.Validate(); err!=nil {
return err
}
}
return nil
}
Then, you can load the files:
var c Config
if err:=LoadConfig("file1",&c.C1); err!=nil {
return err
}
if err:=LoadConfig("file2",&c.C2); err!=nil {
return err
}
...
If there are multiple files loading different parts of the same struct, you can do:
LoadConfig("file1",&c.C3)
LoadConfig("file2",&c.C3)
...
You can simplify this further by defining a slice:
type cfgInfo struct {
fileName string
getCfg func(*Config) interface{}
}
var configs=[]cfgInfo {
{
fileName: "file1",
getCfg: func(c *Config) interface{} {return &c.C1},
},
{
fileName: "file2",
getCfg: func(c *Config) interface{} {return &c.C2},
},
{
fileName: "file3",
getCfg: func(c *Config) interface{} {return &c.C3},
},
...
}
func loadConfigs(cfg *Config) error {
for _,f:=range configs {
if err:=loadConfig(f.fileName,f.getCfg(cfg)); err!=nil {
return err
}
}
return nil
}
Then, loadConfigs would load all the configuration files into cfg.
func main() {
var cfg Config
if err:=loadConfigs(&cfg); err!=nil {
panic(err)
}
...
}
Any configuration that doesn't match this pattern can be dealt with using LoadConfig:
var customConfig1 CustomConfigStruct1
if err:=LoadConfig("customConfigFile1",&customConfig1); err!=nil {
panic(err)
}
cfg.CustomConfig1 = processCustomConfig1(customConfig1)
var customConfig2 CustomConfigStruct2
if err:=LoadConfig("customConfigFile2",&customConfig2); err!=nil {
panic(err)
}
cfg.CustomConfig2 = processCustomConfig2(customConfig2)
I need to create builder (base) and specific builders for each build type.
e.g.
builder for html project
builder for node.js project
builder for python project
builder for java project
….
The main functionality will be like following:
File:Builder.go
interface
type Builder interface {
Build(string) error
}
File: nodebuilder.go
//This is the struct ???? not sure what to put here...
type Node struct {
}
func (n Node) Build(path string) error {
//e.g. Run npm install which build's nodejs projects
command := exec.Command("npm", "install")
command.Dir = “../path2dir/“
Combined, err := command.CombinedOutput()
if err != nil {
log.Println(err)
}
log.Printf("%s", Combined)
}
...
//return new(error)
}
Main assumptions/process:
To start build on each module I need to get the path to it
I need to copy the module to a temp folder
I need to run the build on it (implement the build interface like mvn build npm install etc)
After build was finished zip the module with dep
Copy it to new target folder
Note: beside from the build and the path (that should be handled specifically ) all other functionality are identical
like zip copy
Where should I put the the zip and copy (in the structure) and for example how should I implement them and path them to the builder ?
Should I structure the project differently according the assumptions?
The first principle of SOLID says that a piece of code should have only one responsibility.
Takes the context in, it really makes no sense that any builder to care about the copy and zip part of the build process. It's beyond builder's responsibility. Even using composition (embedding) is not neat enough.
Narrow it down, the Builder's core responsibility is to build the code, as the name suggests. But more specifically, the Builder's responsibility is to build the code at a path. What path? The most idomatic way is the current path, the working dir. This adds two side methods to the interface: Path() string which returns the current path and ChangePath(newPath string) error to change the current path. The implentation would be simple, preserve a single string field as the current path would mostly do the job. And it can easily be extended to some remote process.
If we looks it carefully, there auctually is two build concepts. One is the whole building process, from making temp dir to copy it back, all five steps; the other is the build command, which is the third step of the process.
That is very inspiring. A process is idomatic to be presented as a function, as classic procedural programing would do. So we write a Build function. It serialize all the 5 steps, plain and simple.
Code:
package main
import (
"io/ioutil"
)
//A builder is what used to build the language. It should be able to change working dir.
type Builder interface {
Build() error //Build builds the code at current dir. It returns an error if failed.
Path() string //Path returns the current working dir.
ChangePath(newPath string) error //ChangePath changes the working dir to newPath.
}
//TempDirFunc is what generates a new temp dir. Golang woould requires it in GOPATH, so make it changable.
type TempDirFunc func() string
var DefualtTempDirFunc = func() string {
name,_ := ioutil.TempDir("","BUILD")
return name
}
//Build builds a language. It copies the code to a temp dir generated by mkTempdir
//and call the Builder.ChangePath to change the working dir to the temp dir. After
//the copy, it use the Builder to build the code, and then zip it in the tempfile,
//copying the zip file to `toPath`.
func Build(b Builder, toPath string, mkTempDir TempDirFunc) error {
if mkTempDir == nil {
mkTempDir = DefaultTempDirFunc
}
path,newPath:=b.Path(),mkTempDir()
defer removeDir(newPath) //clean-up
if err:=copyDir(path,newPath); err!=nil {
return err
}
if err:=b.ChangePath(newPath) !=nil {
return err
}
if err:=b.Build(); err!=nil {
return err
}
zipName,err:=zipDir(newPath) // I don't understand what is `dep`.
if err!=nil {
return err
}
zipPath:=filepath.Join(newPath,zipName)
if err:=copyFile(zipPath,toPath); err!=nil {
return err
}
return nil
}
//zipDir zips the `path` dir and returns the name of zip. If an error occured, it returns an empty string and an error.
func zipDir(path string) (string,error) {}
//All other funcs is very trivial.
Most of things are covered in the comments and I am really felling lazy to write all those copyDir/removeDir things. One thing that is not mentioned in the design part is the mkTempDir func. Golang would be unhappy if the code is in /tmp/xxx/ as it is outside the GOPATH, and it would make more trouble to change GOPATH as it will break import path serching, so golang would require a unique function to generate a tempdir inside the GOPATH.
Edit:
Oh, one more thing I forgot to say. It is terribly ugly and irresponsible to handle errors like this. But the idea is there, and more decent error handling mostly requires the usage contents. So do change it yourself, log it, panic or whatever you want.
Edit 2:
You can re-use your npm example as following.
type Node struct {
path string
}
func (n Node) Build(path string) error {
//e.g. Run npm install which build's nodejs project
command := exec.Command("npm", "install")
command.Dir = n.path
Combined, err := command.CombinedOutput()
if err != nil {
log.Println(err)
}
log.Printf("%s", Combined)
return nil
}
func (n *Node) ChangePath(newPath string) error {
n.path = newPath
}
func (n Node) Path() string {
return n.path
}
And to combine it with other language all together:
func main() {
path := GetPathFromInput()
switch GetLanguageName(path) {
case "Java":
Build(&Java{path},targetDirForJava(),nil)
case "Go":
Build(&Golang{path,cgoOptions},targetDirForGo(),GoPathTempDir()) //You can disable cgo compile or something like that.
case "Node":
Build(&Node{path},targetDirForNode(),nil)
}
}
One trick is get language name. GetLanguageName should return the name of the language the code in path is using. This can be done by using ioutil.ReadDir to detect filenames.
Also note that although I made the Node struct very simple and only stores a path field, you can extend it easily. Like in Golang part, you might add build options there.
Edit 3:
About package structure:
First of all, I think literally everything: the Build function, language builders and other util/helpers should be put into a single package. They all work for a single task: build a language. There is no need and hardly any expectation to isolate any piece of code as another (sub)package.
So that means one dir. The remaining is really some very personal style but I will share mine:
I would put the function Build and interface Builder into a file called main.go. If the front-end code is minimal and very readable, I would put them into main.go as well, but if it is long and have some ui-logic, I would put it into a front-end.go or cli.go or ui.go, depending on the auctual code.
Next, for each language, I would create a .go file with the language code. It makes clear where I can check them. Alternatively, if the code is really tiny, it is not a bad idea to put them all together into a builders.go. After all, mordern editors can be more than capable to get definitaion of structs and types.
Finally, all the copyDir, zipDir functions goes to a util.go. That is simple - they are utilities, most time we just don't want to bother them.
Go is not an object-oriented language. This means that by design, you don't necessarily have all the behaviour of a type encapsulated in the type itself. And this is handy when you think that we don't have inheritance.
When you want to build a type upon another type, you use composition instead: a struct can embed other types, and expose their methods.
Let's say that you have a MyZipper type that exposes a Zip(string) method, and a MyCopier that exposes a Copy(string) method:
type Builder struct {
MyZipper
MyCopier
}
func (b Builder) Build(path string) error {
// some code
err := b.Zip(path)
if err != nil {
return err
}
err := b.Copy(path)
if err != nil {
return err
}
}
An this is composition in Go. Going further, you can even embed non-exposed types (e.g. myZipper and myCopier) if you only want them to be called from within the builder package. But then, why embedding them in the first place?
You can chose between several different valid designs for your Go project.
solution 1: Single package exposing multiple Builder types
In this case, you want a builder package, that will expose multiple builders.
zip and copy are two functions defined somewhere in the package, they don't need to be methods attached to a type.
package builder
func zip(zip, args string) error {
// zip implementation
}
func cp(copy, arguments string) error {
// copy implementation
}
type NodeBuilder struct{}
func (n NodeBuilder) Build(path string) error {
// node-specific code here
if err := zip(the, args); err != nil {
return err
}
if err := cp(the, args); err != nil {
return err
}
return nil
}
type PythonBuilder struct{}
func (n PythonBuilder) Build(path string) error {
// python-specific code here
if err := zip(the, args); err != nil {
return err
}
if err := cp(the, args); err != nil {
return err
}
return nil
}
solution 2: single package, single type embedding specific behaviour
Depending on the complexity of the specific behaviour, you may not want to change the whole behaviour of the Build function, but just to inject a specific behaviour:
package builder
import (
"github.com/me/myproj/copier"
"github.com/me/myproj/zipper"
)
type Builder struct {
specificBehaviour func(string) error
}
func (b Builder) Build(path string) error {
if err := specificBehaviour(path); err != nil {
return err
}
if err := zip(the, args); err != nil {
return err
}
if err := copy(the, args); err != nil {
return err
}
return nil
}
func nodeSpecificBehaviour(path string) error {
// node-specific code here
}
func pythonSpecificBehaviour(path string) error {
// python-specific code here
}
func NewNode() Builder {
return Builder{nodeSpecificBehaviour}
}
func NewPython() Builder {
return Builder{pythonSpecificBehaviour}
}
solution 3: one package per specificBuilder
On the other end of the scale, depending on the package granularity you want to use in your project, you may want to have a distinct package for every builder. With this premise, you want to generalise the shared functionality enough to give it a citizenship as package too. Example:
package node
import (
"github.com/me/myproj/copier"
"github.com/me/myproj/zipper"
)
type Builder struct {
}
func (b Builder) Build(path string) error {
// node-specific code here
if err := zipper.Zip(the, args); err != nil {
return err
}
if err := copier.Copy(the, args); err != nil {
return err
}
return nil
}
solution 4: functions!
If you know your builders will be purely functional, meaning they don't need any internal state, then you may want your Builders to be function types intead of interfaces. You will still be able to manipulate them as a single type from the consumer side, if this is what you want:
package builder
type Builder func(string) error
func NewNode() Builder {
return func(string) error {
// node-specific behaviour
if err := zip(the, args); err != nil {
return err
}
if err := copy(the, args); err != nil {
return err
}
return nil
}
}
func NewPython() Builder {
return func(string) error {
// python-specific behaviour
if err := zip(the, args); err != nil {
return err
}
if err := copy(the, args); err != nil {
return err
}
return nil
}
}
I wouldn't go with functions for your particular case, because you will need to solve very different problems with every BUilder, and you will definitely need some state at some point.
... I'll leave you the pleasure to combine together some of these techniques, if you are having a boring afternoon.
Bonus!
Don't be afraid of creating multiple packages, as this will help you design clear boundaries between the types, and take full advantage of encapsulation.
The error keyword is an interface, not a type! You can return nil if you have no errors.
Ideally, you don't define the Builder interface in the builder package: you don't need it. The Builder interface will sit in the consumer package.
Let's go through each question one by one:
1. Where should I put the the zip and copy (in the structure) and for example how should I implement them and path them to the builder ?
An interface does not carry any data (assuming you wanted to implement one from your code). It is just a blueprint an object can implements in order to pass as a more generic type. In this case, if you are not passing Builder type anywhere, the interface is redundant.
2. Should I structure the project differently according the assumptions?
This is my take on the project. I'll explain each part separately after the code:
package buildeasy
import (
"os/exec"
)
// Builder represents an instance which carries information
// for building a project using command line interface.
type Builder struct {
// Manager is a name of the package manager ("npm", "pip")
Manager string
Cmd string
Args []string
Prefn func(string) error
Postfn func(string) error
}
func zipAndCopyTo(path string) error {
// implement zipping and copy to the provided path
return nil
}
var (
// Each manager specific configurations
// are stored as a Builder instance.
// More fields and values can be added.
// This technique goes hand-in-hand with
// `wrapBuilder` function below, which is
// a technique called "functional options"
// which is considered a cleanest approach in
// building API methods.
// https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
NodeConfig = &Builder{
Manager: "npm",
Postfn: zipAndCopyTo,
}
PythonConfig = &Builder{
Manager: "pip",
Postfn: zipAndCopyTo,
}
)
// This enum is used by factory function Create to select the
// right config Builder from the array below.
type Manager int
const (
Npm Manager = iota
Pip
// Yarn
// ...
)
var configs = [...]*Builder{
NodeConfig,
PythonConfig,
// YarnConfig,
}
// wrapBuilder accepts an original Builder and a function that can
// accept a Builder and then assign relevant value to the first.
func wrapBuilder(original *Builder, wrapperfn func(*Builder)) error {
if original != nil {
wrapperfn(original)
return nil
}
return errors.New("Original Builder is nil")
}
func New(manager Manager) *Builder {
builder := new(Builder)
// inject / modify properties of builder with relevant
// value for the manager we want.
wrapBuilder(builder, configs[int(manager)])
})
return builder
}
// Now you can have more specific methods like to install.
// notice that it doesn't matter what this Builder is for.
// All information is contained in it already.
func (b *Builder) Install(pkg string) ([]byte, error) {
b.Cmd = "install"
// if package is provided, push its name to the args list
if pkg != "" {
b.Args = append([]string{pkg}, b.Args...)
}
// This emits "npm install [pkg] [args...]"
cmd := exec.Command(b.Manager, (append([]string{b.Cmd}, b.Args...))...)
// default to executing in the current directory
cmd.Dir = "./"
combined, err := cmd.CombinedOutput()
if err != nil {
return nil, err
}
return combined, nil
}
func (b *Builder) Build(path string) error {
// so default the path to a temp folder
if path == "" {
path = "path/to/my/temp"
}
// maybe prep the source directory?
if err := b.Prefn(path); err != nil {
return err
}
// Now you can use Install here
output, err := b.Install("")
if err != nil {
return err
}
log.Printf("%s", output)
// Now zip and copy to where you want
if err := b.Postfn(path); err != nil {
return err
}
return nil
}
Now this Builder is generic enough to handle most build commands. Notice Prefn and Postfn fields. These are hook functions you can run before and after the command runs within Build. Prefn can check if, say, the package manager is installed and install it if it's not (or just return an error). Postfn can run your zip and copy operations, or any clean up routine. Here's a usecase, provided superbuild is our fictional package name and the user is using it from outside:
import "github.com/yourname/buildeasy"
func main() {
myNodeBuilder := buildeasy.New(buildeasy.NPM)
myPythonBuilder := buildeasy.New(buildeasy.PIP)
// if you wanna install only
myNodeBuilder.Install("gulp")
// or build the whole thing including pre and post hooks
myPythonBuilder.Build("my/temp/build")
// or get creative with more convenient methods
myNodeBuilder.GlobalInstall("gulp")
}
You can predefine a few Prefns and Postfns and make them available as option for the user of your program, assuming it's a command line program, or if it's a library, have the user write them herself.
wrapBuilder function
There are a few techniques used in constructing an instance in Go. First, parameters can be passed into a constructor function (this code is for explanation only and not to be used):
func TempNewBuilder(cmd string) *Builder {
builder := new(Builder)
builder.Cmd = cmd
return builder
}
But this approach is very ad-hoc because it is impossible to pass arbitrary values to configure the returned *Builder. A more robust approach is to pass a config instance of *Builder:
func TempNewBuilder(configBuilder *Builder) *Builder {
builder := new(Builder)
builder.Manager = configBuilder.Manager
builder.Cmd = configBuilder.Cmd
// ...
return builder
}
By using wrapBuilder function, you can write a function to handle (re)assigning of values of an instance:
func TempNewBuilder(builder *Builder, configBuilderFn func(*Builder)) *Builder {
configBuilderFn(builder)
}
Now you can pass in any function for configBuilderFn to configure your *Builder instance.
To read more, see https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis.
configs array
configs array goes hand-in-hand with the enum of Manager constants. Take a look at New factory function. the enum constant manager passed in pass the parameter is type Manager which is just an int underneath. This means all we had to do is access configs using the manager as the index in wrapBuilder:
wrapBuilder(builder, configs[int(manager)])
For instance, if manager == Npm,configs[int(manager)] will return NodeConfig from configs array.
Structuring package(s)
At this point, it is fine to have zip and copy functions to live in the same package as Build the way I did. There's little use in prematurely optimizing anything or worry about that until you have to. That will only introduce more complexity than you'd want. Optimization comes consistently as you develop the code.
If you feel like structuring the project early is important, you can do it based on the semantic of your API. For instance, to create a new *Builder, it is quite intuitive for the user to call a factory function New or Create from a subpackage buildeasy/builder:
// This is a user using your `buildeasy` package
import (
"github.com/yourname/buildeasy"
"github.com/yourname/buildeasy/node"
"github.com/yourname/buildeasy/python"
)
var targetDir = "path/to/temp"
func main() {
myNodeBuilder := node.New()
myNodeBuilder.Build(targetDir)
myPythonBuilder := python.New()
myPythonBuilder.Install("tensorflow")
}
Another more verbose approach is to include the semantic as part of the function's name, which is also used in Go's standard packages:
myNodeBuilder := buildeasy.NewNodeBuilder()
myPythonBuilder := buildeasy.NewPipBuilder()
// or
mySecondNodeBuilder := buildeasy.New(buildeasy.Yarn)
In Go's standard packages, verbose functions and methods are common. That's because it normally structure sub-packages (subdirectories) for more specific utilities such as path/filepath, which contains utility functions related to file path manipulation while keeping path's API basic and clean.
Coming back to your project, I would keep most common, more generic functions at the top level directory/package. This is how I would tackle the structure:
buildeasy
├── buildeasy.go
├── python
│ └── python.go
└── node/
└── node.go
While package buildeasy contains functions like NewNodeBuilder, NewPipBuilder, or just New that accepts additional options (like the above code), in subpackage buildeasy/node, for instance, can look like this:
package node
import "github.com/yourname/buildeasy"
func New() *buildeasy.Builder {
return buildeasy.New(buildeasy.Npm)
}
func NewWithYarn() *buildeasy.Builder {
return buildeasy.New(buildeasy.Yarn)
}
// ...
or buildeasy/python:
package python
import "github.com/yourname/buildeasy"
func New() *buildeasy.Builder {
return buildeasy.New(buildeasy.Pip)
}
func NewWithEasyInstall() *buildeasy.Builder {
return buildeasy.New(buildeasy.EasyInstall)
}
// ...
Note that in the subpackages you never have to call buildeasy.zipAndCopy because it is a private function that is lower level than the node and python subpackages should care. these subpackages act like another layer of API calling buildeasy's functions and passing in some specific values and configurations that make life easier for the user of its API.
Hope this makes sense.
I've been having some issues loading html templates using the Gin framework through the r.HTMLRender setting.
It would seem that the templates are not being found.
I have tried to use the following helpers:
GinHTMLRender: https://gist.github.com/madhums/4340cbeb36871e227905
EZ Gin Template: https://github.com/michelloworld/ez-gin-template
Neither of these seem to be working when setting the default path for templates (in my case app/views); for the purposes of getting to this to work my html template file structure looks like this:
/workspace
|-app
|-views
|-layouts
|-application.html
|-test.html
Here is a sample of the Gin loading code:
import (
"github.com/gin-contrib/location"
"github.com/gin-gonic/gin"
"fmt"
"os"
"github.com/gin-gonic/contrib/static"
"github.com/michelloworld/ez-gin-template"
)
//CORSMiddleware ...
func CORSMiddleware() gin.HandlerFunc {
/** CORS middleware **/
}
func Router() {
if os.Getenv("ENVIRONMENT") == "production" {
gin.SetMode(gin.ReleaseMode)
}
// Initialize Gin object
r := gin.Default()
// Cors Middleware
r.Use(CORSMiddleware())
// Rate limiting
rl, err := helpers.RateLimiterMiddleware()
if err != nil {
panic("Rate Limiting Initialization error")
}
r.Use(rl)
// Asset provision
r.Use(static.ServeRoot("/public","app/assets"))
// Get URL information
r.Use(location.Default())
// Attempt with EZ Template, fails
// I ge this error: "runtime error: invalid memory address or nil pointer dereference" when calling c.HTML(...)
render := eztemplate.New()
render.TemplatesDir = "app/views/" // default
render.Layout = "layouts/application" // default
render.Ext = ".html" // default
render.Debug = true // default
r.HTMLRender = render.Init()
// Attempt with GinHTMLRender, fails
// I get this error: https://gist.github.com/madhums/4340cbeb36871e227905#file-gin_html_render-go-L110
/*
htmlRender := GinHTMLRender.New()
htmlRender.TemplatesDir = "app/views/"
htmlRender.Debug = gin.IsDebugging()
htmlRender.Layout = "layouts/application"
log.Println("Dir:"+htmlRender.TemplatesDir)
r.HTMLRender = htmlRender.Create()*/
/** Some Routes **/
// Start web listener
r.Run(":8009") // listen and serve on 0.0.0.0:8080
}
The corresponding render call is code is the following:
/* c is of type *gin.Context */
c.HTML(200, "test", "")
For some reason it seems like the r.HTMLRender function is not taking into account the template path; I have attempted doing this:
_, err := template.ParseFiles("app/views/test.html")
if err != nil {
log.Println("Template Error")
} else {
log.Println("No Template Error")
}
This code consistently displays "No Template Error", which leads me to believe that the HTMLRender assignment is not considering the TemplatesDir set variable.
I've been stuck with this issue for some time, and I am not entirely sure how to get it resolved.
Any help getting this to work would be greatly appreciated.
After doing some further research I found the source of my problem with EZ Gin Template.
I'm hoping this helps anyone experiencing the same issue.
After taking a deeper look at the helper code, I realized that the template matching pattern is strict and does not search for files recursively; ie. it expects a specific file structure to find template files:
In the default setting, EZ Gin Template requires the following file structure to work:
/workspace
- app
- views
- layouts
- some_layout.html
- some_dir
- template_file.html
- _partial_template.html
- partials
- _some_other_partial.html
In order to allow for other file patterns, the helper set of functions needs to be modified.
In my case, I forked the helper code locally to allow matching 1st level template files:
func (r Render) Init() Render {
globalPartials := r.getGlobalPartials()
layout := r.TemplatesDir + r.Layout + r.Ext
// Match multiple levels of templates
viewDirs, _ := filepath.Glob(r.TemplatesDir + "**" + string(os.PathSeparator) + "*" + r.Ext)
// Added the following two lines to match for app/views/some_file.html as well as files on the **/*.html matching pattern
tmp, _ := filepath.Glob(r.TemplatesDir + "*" + r.Ext)
viewDirs = append(viewDirs, tmp...)
// Can be extended by replicating those two lines above and adding search paths within the base template path.
fullPartialDir := filepath.Join(r.TemplatesDir + r.PartialDir)
for _, view := range viewDirs {
templateFileName := filepath.Base(view)
//skip partials
if strings.Index(templateFileName, "_") != 0 && strings.Index(view, fullPartialDir) != 0 {
localPartials := r.findPartials(filepath.Dir(view))
renderName := r.getRenderName(view)
if r.Debug {
log.Printf("[GIN-debug] %-6s %-25s --> %s\n", "LOAD", view, renderName)
}
allFiles := []string{layout, view}
allFiles = append(allFiles, globalPartials...)
allFiles = append(allFiles, localPartials...)
r.AddFromFiles(renderName, allFiles...)
}
}
return r
}
I have not tried a similar solution with GinHTMLRenderer, but I expect that the issue might likely be related to it in terms of the expected file structure.
You can also bind the templates into the code. The jessevdk/go-assets-builder will generate a go file that contains assets within the specified directory. Make sure that the file generated is located where the main package is. Gin also provided this as an example in their Documentation For more Info. It will also include subfolders and its files (i. e. assets) in the binary.
Get The generator tool:
go get github.com/jessevdk/go-assets-builder
Generate:
# go-assets-builder <dir> -o <generated file name>
go-assets-builder app -o assets.go
Note that <generated file name> can also be like cmd/client/assets.go to specify the destination of the file to be generated.
Load Template:
package main
// ... imports
func main() {
r := gin.New()
t, err := loadTemplate()
if err != nil {
panic(err)
}
r.SetHTMLTemplate(t)
r.GET("/", func(c *gin.Context) {
c.HTML(http.StatusOK, "app/views/layouts/application.html", nil)
})
r.GET("/test", func(c *gin.Context) {
c.HTML(http.StatusOK, "app/views/test.html", nil)
})
r.Run(":8080")
}
// loadTemplate loads templates embedded by go-assets-builder
func loadTemplate() (*template.Template, error) {
t := template.New("")
// Assets is the templates
for name, file := range Assets.Files {
if file.IsDir() || !strings.HasSuffix(name, ".html") {
continue
}
h, err := ioutil.ReadAll(file)
if err != nil {
return nil, err
}
t, err = t.New(name).Parse(string(h))
if err != nil {
return nil, err
}
}
return t, nil
}
Here is how I do it. This walks through the directory and collects the files marked with my template suffix which is .html & then I just include all of those. I haven't seen this answer anywhere so I thought Id post it.
// START UP THE ROUTER
router := gin.Default()
var files []string
filepath.Walk("./views", func(path string, info os.FileInfo, err error) error {
if strings.HasSuffix(path, ".html") {
files = append(files, path)
}
return nil
})
router.LoadHTMLFiles(files...)
// SERVE STATICS
router.Use(static.Serve("/css", static.LocalFile("./css", true)))
router.Use(static.Serve("/js", static.LocalFile("./js", true)))
router.Use(static.Serve("/images", static.LocalFile("./images", true)))
routers.LoadBaseRoutes(router)
routers.LoadBlog(router)
router.Run(":8080")
Now they don't have to all be nested at the exact depth like the other suggestions ... the file structure can be uneven
I have a runtime config instance that I need in other parts of the app, but it can only be created in main(). Ideally I would like to avoid using global variables.
// main.go
type RuntimeConfig struct {
db *DatabaseInstance
app_name string
... etc ...
}
func main() {
dbInstance = ConnectToDB(...args) // returns *DatabaseInstance
runtimeConfig := *Config{
dbInstance,
"My app",
... etc ...
}
}
// elsewhere.go
func SomeUtilityFuncThatNeedsRuntime(i int) int {
runtime := GetRuntime() // imaginary, magical runtime getter
db := runtime.DatabaseInstance
appName := runtime.appName
db.Save(appName, db, ...)
return i + 1
}
Currently it's impossible to create anonymous util functions that could really benefit from having access to certain config variables. If the variables were basic types (like a string or int), I would probably just hard-code them in. However, a field like dbInstance requires a specific instance of a connected database.
This looks to me like a use case for the singleton pattern: your RuntimeConfig is a structure that should be initialized, exactly one instance of it should exist, and it should be possible to access it.
Create configuration package with private variable and public functions (pseudo code):
package configuration
type Configuration struct {
}
var config *Configuration = nil
func GetConfig() *Configuration {
return config
}
func configLoad(filePath string) error {
config = new(Configuration)
// load your config from file, fill config structure
return nil
}
func NewConfig(flags models.ConfigFlags) (*Configuration, error) {
err := configLoad(flags.Flagconfiguration) // Path of config file.
if err != nil {
return nil, err
}
return config, nil
}
Then in your main.go initialize config:
func main() {
config, err := configuration.NewConfig(FlagsParameters)
// use this config variable in main package
}
In other packages use:
config := configuration.Config()
As an alternative, you can implement singleton pattern (but I like it less)
type Configuration struct {
}
var config *Configuration
var once sync.Once
func GetConfig() *Configuration {
once.Do(func() {
// init your config here. This code will executed once and thread safe
})
return config
}
Upon getting session information of type map[string]interface{} with this.GetSession("session_key"), I did have to explicitly set the context and type assert the session like this in order to explicitly pass the data to the template.
// Get the session
profile := this.GetSession("profile")
// Have to add data to the template's context
this.Data["nickname"] = profile.(map[string]interface{})["nickname"].(string)
this.Data["picture"] = profile.(map[string]interface{})["picture"].(string)
// Render template
this.TplNames = "user.html"
The session data (type map[string]interface{}) looks like this:
{"nickname": "joe", "picture": "urltotheimg"}
However, according to the Beego's session doc, it looks like the session is passed implicitly without any need of type assertions or context passing (the template has immediate access to session values i.e. {{.nickname}} and {{.picture}})
This is the controller setting the session before redirecting to /user
// Inherit beego's base controller
type MyController struct {
beego.Controller
}
func (this *MyController) Get() {
// code for getting token here
// Getting the User information
client := conf.Client(oauth2.NoContext, token)
resp, err := client.Get("https://" + domain + "/userinfo")
if err != nil {
this.Redirect("/error", 500)
return
}
// Reading the body for user's information
raw, err := ioutil.ReadAll(resp.Body)
defer resp.Body.Close()
if err != nil {
this.Redirect("/error", 500)
return
}
// Unmarshalling the JSON of the Profile
var profile map[string]interface{}
if err := json.Unmarshal(raw, &profile); err != nil {
this.Redirect("/error", 500)
return
}
// Saving the information to the session.
this.SetSession("profile", profile)
// redirect to /user
this.Redirect("/user", 301)
}
This is the controller of "/user"
type UserController struct {
beego.Controller
}
func (this *UserController) Get() {
// get the saved session
profile := this.GetSession("profile")
// without setting the template data here, the session data won't be
// available in user.html
this.Data["nickname"] = profile.(map[string]interface{})["nickname"].(string)
this.Data["picture"] = profile.(map[string]interface{})["picture"].(string)
this.TplNames = "user.html"
}
Only this then I can map the template to the data like this:
<img src="{{ .picture }}">
<p>Hello, {{ .nickname }}</p>
I'm quite sure it's necessary to set the template data. I'm just not sure why the above doc didn't do that.
Any help would be appreciated.
I just tried running the Beego quickstart project and ran it successfully.
Make sure you have both the beego and bee installed. After creating a new project with bee new projectname make sure you edit the projectname/conf/app.conf file and add the sessionon = true:
appname = quickstart
httpport = 8080
runmode = dev
sessionon = true
I created a redirect controller like:
type RedirectController struct {
beego.Controller
}
func (c *RedirectController) Get() {
profile := make(map[string]interface{})
profile["nickname"] = "User's Nickname"
profile["picture"] = "/path/to/img.jpg"
c.SetSession("profile", profile)
c.Redirect("/", 301)
}
The main controller:
type MainController struct {
beego.Controller
}
func (c *MainController) Get() {
profile := c.GetSession("profile")
c.Data["nickname"] = profile.(map[string]interface{})["nickname"]
c.Data["picture"] = profile.(map[string]interface{})["picture"]
c.TplNames = "index.tpl"
}
My index.tpl file:
<p>Nickname: {{.nickname}}</p>
<p>Picture: {{.picture}}</p>
And the router:
func init() {
beego.Router("/", &controllers.MainController{})
beego.Router("/redirect", &controllers.RedirectController{})
}
I would also recommend you to use a structure to store the profile values like:
// Define struct.
type Profile struct{
Nickname string
Picture string
}
// Save it for template rendering.
this.Data["profile"] = &Profile{Nickname:"astaxie", Picture:"img.jpg"}
// And render it like this:
Nickname: {{.profile.Nickname}}
Picture: {{.profile.Picture}}
Make sure to read this to understand how template rendering is done. I hope this is what you were asking for, if not, please edit your question and add more helpful information and I will edit this answer.