I want to be able to unmarshal yaml files less rigidly. That is, my library has a predefined number of options the yaml file must have. Then, the user should be able to extend this to include any custom options.
Here is what I have
package main
import (
"net/http"
"yamlcms"
"github.com/julienschmidt/httprouter"
)
type Page struct {
*yamlcms.Page
Title string
Date string
}
func getBlogRoutes() {
pages := []*Page{}
yamlcms.ReadDir("html", pages)
}
// This section is a work in progress, I only include it for loose context
func main() {
router := httprouter.New()
//blogRoutes := getBlogRoutes()
//for _, blogRoute := range *blogRoutes {
// router.Handle(blogRoute.Method, blogRoute.Pattern,
// func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {})
//}
http.ListenAndServe(":8080", router)
}
Here is the yamlcms package:
package yamlcms
import (
"io/ioutil"
"os"
"strings"
"gopkg.in/yaml.v2"
)
type Page struct {
Slug string `yaml:"slug"`
File string `yaml:"file"`
}
func (page *Page) ReadFile(file string) (err error) {
fileContents, err := ioutil.ReadFile(file)
if err != nil {
return
}
err = yaml.Unmarshal(fileContents, &page)
return
}
func isYamlFile(fileInfo os.FileInfo) bool {
return !fileInfo.IsDir() && strings.HasSuffix(fileInfo.Name(), ".yaml")
}
func ReadDir(dir string, pages []*Page) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
pages[i].ReadFile(fileInfo.Name())
}
}
return
}
There is a compiler issue here:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.Page in argument to yamlcms.ReadDir
My main intent in this question is to learn the idiomatic way of doing this kind of thing in Go. Other 3rd-party solutions may exist but I am not immediately interested in them because I have problems like this frequently in Go having to do with inheritance, etc. So along the lines of what I've presented, how can I best (idiomatically) accomplish what I am going for?
EDIT:
So I've made some changes as suggested. Now I have this:
type FileReader interface {
ReadFile(file string) error
}
func ReadDir(dir string, pages []*FileReader) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
(*pages[i]).ReadFile(fileInfo.Name())
}
}
return
}
However, I still get a similar compiler error:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.FileReader in argument to yamlcms.ReadDir
Even though main.Page should be a FileReader because it embeds yamlcms.Page.
EDIT: I forgot that slices of interfaces don't work like that. You'd need to allocate a new slice, convert all pages to FileReaders, call the function, and convert them back.
Another possible solution is refactoring yamlcms.ReadDir to return the contents of the files, so that they could be unmarshaled later:
// In yamlcms.
func ReadYAMLFilesInDir(dir string) ([][]byte, error) { ... }
// In client code.
files := yamlcms.ReadYAMLFilesInDir("dir")
for i := range pages {
if err := yaml.Unmarshal(files[i], &pages[i]); err != nil { return err }
}
The original answer:
There are no such things as inheritance or casting in Go. Prefer composition and interfaces in your designs. In your case, you can redefine your yamlcms.ReadDir to accept an interface, FileReader.
type FileReader interface {
ReadFile(file string) error
}
Both yamlcms.Page and main.Page will implement this, as the latter embeds the former.
Related
I am working with lot of config files. I need to read all those individual config file in their own struct and then make one giant Config struct which holds all other individual config struct in it.
Let's suppose if I am working with 3 config files.
ClientConfig deals with one config file.
DataMapConfig deals with second config file.
ProcessDataConfig deals with third config file.
I created separate class for each of those individual config file and have separate Readxxxxx method in them to read their own individual config and return struct back. Below is my config.go file which is called via Init method from main function after passing path and logger.
config.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type Config struct {
ClientMapConfigs ClientConfig
DataMapConfigs DataMapConfig
ProcessDataConfigs ProcessDataConfig
}
func Init(path string, logger log.Logger) (*Config, error) {
var err error
clientConfig, err := ReadClientMapConfig(path, logger)
dataMapConfig, err := ReadDataMapConfig(path, logger)
processDataConfig, err := ReadProcessDataConfig(path, logger)
if err != nil {
return nil, err
}
return &Config{
ClientMapConfigs: *clientConfig,
DataMapConfigs: *dataMapConfig,
ProcessDataConfigs: *processDataConfig,
}, nil
}
clientconfig.go
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type ClientConfig struct {
.....
.....
}
const (
ClientConfigFile = "clientConfigMap.json"
)
func ReadClientMapConfig(path string, logger log.Logger) (*ClientConfig, error) {
files, err := utilities.FindFiles(path, ClientConfigFile)
// read all the files
// do some validation on all those files
// deserialize them into ClientConfig struct
// return clientconfig object back
}
datamapconfig.go
Similar style I have for datamapconfig too. Exactly replica of clientconfig.go file but operating on different config file name and will return DataMapConfig struct back.
processdataConfig.go
Same thing as clientconfig.go file. Only difference is it will operate on different config file and return ProcessDataConfig struct back.
Problem Statement
I am looking for ideas where this above design can be improved? Is there any better way to do this in golang? Can we use interface or anything else which can improve the above design?
If I have let's say 10 different files instead of 3, then do I need to keep doing above same thing for remaining 7 files? If yes, then the code will look ugly. Any suggestions or ideas will greatly help me.
Update
Everything looks good but I have few questions as I am confuse on how can I achieve those with your current suggestion. On majority of my configs, your suggestion is perfect but there are two cases on two different configs where I am confuse on how to do it.
Case 1 After deserializing json into original struct which matches json format, I make another different struct after massaging that data and then I return that struct back.
Case 2 All my configs have one file but there are few configs which have multiple files in them and the number isn't fixed. So I pass regex file name and then I find all the files starting with that regex and then loop over all those files one by one. After deserializing each json file, I start populating another object and keep populating it until all files have been deserialized and then make a new struct with those objects and then return it.
Example of above scenarios:
Sample case 1
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type CustomerManifest struct {
CustomerManifest map[int64]Site
}
type CustomerConfigs struct {
CustomerConfigurations []Site `json:"customerConfigurations"`
}
type Site struct {
....
....
}
const (
CustomerConfigFile = "abc.json"
)
func ReadCustomerConfig(path string, logger log.Logger) (*CustomerManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, CustomerConfigFile)
if err != nil {
return nil, err
}
var customerConfig CustomerConfigs
// there is only file for this config so loop will run once
for _, file := range files {
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, &customerConfig)
if err != nil {
return nil, err
}
}
customerConfigIndex := BuildIndex(customerConfig, logger)
return &CustomerManifest{CustomerManifest: customerConfigIndex}, nil
}
func BuildIndex(customerConfig CustomerConfigs, logger log.Logger) map[int64]Site {
...
...
}
As you can see above in sample case 1, I am making CustomerManifest struct from CustomerConfigs struct and then return it instead of returning CustomerConfigs directly.
Sample case 2
package config
import (
"encoding/json"
"fmt"
"io/ioutil"
"github.com/david/internal/utilities"
)
type StateManifest struct {
NotionTemplates NotionTemplates
NotionIndex map[int64]NotionTemplates
}
type NotionMapConfigs struct {
NotionTemplates []NotionTemplates `json:"notionTemplates"`
...
}
const (
// there are many files starting with "state-", it's not fixed number
StateConfigFile = "state-*.json"
)
func ReadStateConfig(path string, logger log.Logger) (*StateManifest, error) {
// I try to find all the files with my below utility method.
// Work with single file name and also with regex name
files, err := utilities.FindFiles(path, StateConfigFile)
if err != nil {
return nil, err
}
var defaultTemp NotionTemplates
var idx = map[int64]NotionTemplates{}
// there are lot of config files for this config so loop will run multiple times
for _, file := range files {
var notionMapConfig NotionMapConfigs
body, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
err = json.Unmarshal(body, ¬ionMapConfig)
if err != nil {
return nil, err
}
for _, tt := range notionMapConfig.NotionTemplates {
if tt.IsProcess {
defaultTemp = tt
} else if tt.ClientId > 0 {
idx[tt.ClientId] = tt
}
}
}
stateManifest := StateManifest{
NotionTemplates: defaultTemp,
NotionIndex: idx,
}
return &stateManifest, nil
}
As you can see above in my both the cases, I am making another different struct after deserializing is done and then I return that struct back but as of now in your current suggestion I think I won't be able to do this generically because for each config I do different type of massaging and then return those struct back. Is there any way to achieve above functionality with your current suggestion? Basically for each config if I want to do some massaging, then I should be able to do it and return new modified struct back but for some cases if I don't want to do any massaging then I can return direct deserialize json struct back. Can this be done generically?
Since there are config which has multiple files in them so that is why I was using my utilities.FindFiles method to give me all files basis on file name or regex name and then I loop over all those files to either return original struct back or return new struct back after massaging original struct data.
You can use a common function to load all the configuration files.
Assume you have config structures:
type Config1 struct {...}
type Config2 struct {...}
type Config3 struct {...}
You define configuration validators for those who need it:
func (c Config1) Validate() error {...}
func (c Config2) Validate() error {...}
Note that these implement a Validatable interface:
type Validatable interface {
Validate() error
}
There is one config type that includes all these configurations:
type Config struct {
C1 Config1
C2 Config2
C3 Config3
...
}
Then, you can define a simple configuration loader function:
func LoadConfig(fname string, out interface{}) error {
data, err:=ioutil.ReadFile(fname)
if err!=nil {
return err
}
if err:=json.Unmarshal(data,out); err!=nil {
return err
}
// Validate the config if necessary
if validator, ok:=out.(Validatable); ok {
if err:=validator.Validate(); err!=nil {
return err
}
}
return nil
}
Then, you can load the files:
var c Config
if err:=LoadConfig("file1",&c.C1); err!=nil {
return err
}
if err:=LoadConfig("file2",&c.C2); err!=nil {
return err
}
...
If there are multiple files loading different parts of the same struct, you can do:
LoadConfig("file1",&c.C3)
LoadConfig("file2",&c.C3)
...
You can simplify this further by defining a slice:
type cfgInfo struct {
fileName string
getCfg func(*Config) interface{}
}
var configs=[]cfgInfo {
{
fileName: "file1",
getCfg: func(c *Config) interface{} {return &c.C1},
},
{
fileName: "file2",
getCfg: func(c *Config) interface{} {return &c.C2},
},
{
fileName: "file3",
getCfg: func(c *Config) interface{} {return &c.C3},
},
...
}
func loadConfigs(cfg *Config) error {
for _,f:=range configs {
if err:=loadConfig(f.fileName,f.getCfg(cfg)); err!=nil {
return err
}
}
return nil
}
Then, loadConfigs would load all the configuration files into cfg.
func main() {
var cfg Config
if err:=loadConfigs(&cfg); err!=nil {
panic(err)
}
...
}
Any configuration that doesn't match this pattern can be dealt with using LoadConfig:
var customConfig1 CustomConfigStruct1
if err:=LoadConfig("customConfigFile1",&customConfig1); err!=nil {
panic(err)
}
cfg.CustomConfig1 = processCustomConfig1(customConfig1)
var customConfig2 CustomConfigStruct2
if err:=LoadConfig("customConfigFile2",&customConfig2); err!=nil {
panic(err)
}
cfg.CustomConfig2 = processCustomConfig2(customConfig2)
I use a function that requires a filename as a parameter (of type string). It works fine when providing the filename.
I would like to embed this file in my binary. I can then have the contents as []byte or string but that's not useful. I can also fave it as embed.FS but my understanding is that this is an abstraction that can be used by some functions only.
What I would need is the ability to present this embedded file as a filename (a string) that would then be used by the underlying function to open the (embedded) file.
Is this possible?
Filename that Key accept as string argument is only abstraction on ioutil.ReadFile, see auth.go
What you can do is implement ssh.Auth yourself, here is small example.
package main
import (
_ "embed"
"fmt"
"github.com/melbahja/goph"
"golang.org/x/crypto/ssh"
)
//go:embed id_rsa
var privateKey []byte
func main() {
auth, err := Auth(privateKey, []byte("foobar"))
fmt.Println(auth, err)
}
func Auth(privateKey, pass []byte) (goph.Auth, error) {
signer, err := Singer(privateKey, pass)
if err != nil {
return nil, err
}
return goph.Auth{
ssh.PublicKeys(signer),
}, nil
}
func Singer(privateKey, pass []byte) (ssh.Signer, error) {
if len(pass) != 0 {
return ssh.ParsePrivateKeyWithPassphrase(privateKey, pass)
}
return ssh.ParsePrivateKey(privateKey)
}
I'm writing a chat bot in Go and wondering how can I avoid a long switch-case statement similar to this one:
switch {
// #bot search me HMAC
case strings.Contains(message, "search me"):
query := strings.Split(message, "search me ")[1]
return webSearch(query), "html"
// #bot thesaurus me challenge
case strings.Contains(message, "thesaurus me"):
query := strings.Split(message, "thesaurus me ")[1]
return synonyms(query), "html"
Should I define those handlers each in a separate package or should I just use structs and interfaces? Which method will allow me to have a good structure, avoid switch-case and let external developers to easier create handlers?
I think packages will be a better choice but I'm not sure how to register the handlers with the main bot. Would appreciate an example.
You could use a map[string]command similar to how the net/http package registers handlers. Something akin to this:
https://play.golang.org/p/9YzHyLodAQ
package main
import (
"fmt"
"errors"
)
type BotFunc func(string) (string, error)
type BotMap map[string]BotFunc
var Bot = BotMap{}
func (b BotMap) RegisterCommand(command string, f BotFunc) error {
if _, exists := b[command]; exists {
return errors.New("command already exists")
}
b[command] = f
return nil
}
func (b BotMap) Execute(statement string) (string, error) {
// parse out command and query however you choose (not this way obviously)
command := statement[:9]
query := statement[10:]
return b.ExecuteQuery(command, query)
}
func (b BotMap) ExecuteQuery(command, query string) (string, error) {
if com, exists := b[command]; exists {
return com(query)
}
return "", errors.New("command doesn't exist")
}
func main() {
err := Bot.RegisterCommand("search me", func(query string) (string, error) {
fmt.Println("search", query)
return "searched", nil
})
if err != nil {
fmt.Println(err)
return
}
err = Bot.RegisterCommand("thesaurus me", func(query string) (string, error) {
fmt.Println("thesaurus", query)
return "thesaurused", nil
})
if err != nil {
fmt.Println(err)
return
}
result, err := Bot.Execute("search me please")
if err != nil {
fmt.Println(err)
return
}
fmt.Println(result)
}
Obviously there's a lot of checks missing here, but this is the basic idea.
I've a server in golang who handle folder path like that :
fs := http.FileServer(http.Dir("./assets"))
http.Handle("/Images/", fs)
http.ListenAndServe(":8000", nil)
But in this folder there are privates images, and it shouldn't be possible to access files. So how can i secure image access and prevent anybody to access content of folder.
like that for example :
If you want to block a directory using http package, maybe this will be useful to you :
https://groups.google.com/forum/#!topic/golang-nuts/bStLPdIVM6w
package main
import (
"net/http"
"os"
)
type justFilesFilesystem struct {
fs http.FileSystem
}
func (fs justFilesFilesystem) Open(name string) (http.File, error) {
f, err := fs.fs.Open(name)
if err != nil {
return nil, err
}
return neuteredReaddirFile{f}, nil
}
type neuteredReaddirFile struct {
http.File
}
func (f neuteredReaddirFile) Readdir(count int) ([]os.FileInfo, error) {
return nil, nil
}
func main() {
fs := justFilesFilesystem{http.Dir("/tmp/")}
http.ListenAndServe(":8080", http.FileServer(fs))
}
A little wrapper over FileServer() solves your problem, now you have to add some sort of logic to do Authorization, it looks like you have unique names, that's good, so I just filter the image name for you creating a map of names, now you can add something more dynamic like a key/store(memcached, redis. etc.) Hope you can follow the comments
package main
import (
"log"
"net/http"
"strings"
)
// put the allowed hashs or keys here
// you may consider put them in a key/value store
//
var allowedImages = map[string]bool{
"key-abc.jpg": true,
"key-123.jpg": true,
}
func main() {
http.Handle("/Images/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// here we can do any kind of checking, in this case we'll just split the url and
// check if the image name is in the allowedImages map, we can check in a DB or something
//
parts := strings.Split(r.URL.Path, "/")
imgName := parts[len(parts)-1]
if _, contains := allowedImages[imgName]; !contains { // if the map contains the image name
log.Printf("Not found image: %q path: %s\n", imgName, r.URL.Path)
// if the image is not found we write a 404
//
// Bonus: we don't list the directory, so nobody can know what's inside :)
//
http.NotFound(w, r)
return
}
log.Printf("Serving allowed image: %q\n", imgName)
fileServer := http.StripPrefix("/Images/", http.FileServer(http.Dir("./assets")))
fileServer.ServeHTTP(w, r) // StripPrefix() and FileServer() return a Handler that implements ServerHTTP()
}))
http.ListenAndServe(":8000", nil)
}
https://play.golang.org/p/ehrd_AWXim
I'm still quite new to Go and I was surprised to not be able to use the subtype of an embedded interface.
Here is a small example to explain what I mean:
func test(sl bufio.ReadWriter){
// cannot use sl(type bufio.ReadWriter) as type bufio.Reader in function argument
readStuff(sl)
[...]
writeStuff(sl) // same kind of error
}
func readStuff(sl bufio.Reader){
[...]
}
As every interface have the same memory layout and ReadWriter is a Reader and a Writer, I was expecting this code to work.
I did try to convert the interface type with:
readStuff(sl.(buffio.Reader))
But it doesn't work either. So I've got two questions:
Why doesn't it work?
What's the go philosophy about that problem?
They're different types. However, a bufio.ReadWriter contains a pointer to both a bufio.Reader type and a bufio.Writer type as elements of its struct. So passing the correct one should be easy enough. Try this:
func test(sl bufio.ReadWriter){
readStuff(sl.Reader)
[...]
writeStuff(sl.Writer)
}
// Changed this bufio.Reader to a pointer receiver
func readStuff(sl *bufio.Reader) {
[...]
}
bufio.ReadWriter is a concrete type, not an interface. However, it does satisfy an interface (io.ReadWriter) so it can be assigned to a variable/function argument of an appropriate interface type. Then it works the way you may have anticipated (your code actually doesn't use any interfaces):
package main
import (
"bufio"
"bytes"
"fmt"
"io"
"log"
)
func readStuff(r io.Reader) {
b := make([]byte, 10)
n, err := r.Read(b)
if err != nil && err != io.EOF {
log.Fatal(err)
}
fmt.Printf("readStuff: %q\n", b[:n])
}
func writeStuff(w io.Writer) {
b := []byte("written")
n, err := w.Write(b)
if n != len(b) {
log.Fatal("Short write")
}
if err != nil {
log.Fatal(err)
}
}
func test(rw io.ReadWriter) {
readStuff(rw)
writeStuff(rw)
}
func main() {
r := io.Reader(bytes.NewBufferString("source"))
var uw bytes.Buffer
w := io.Writer(&uw)
rw := bufio.NewReadWriter(bufio.NewReader(r), bufio.NewWriter(w))
test(rw)
rw.Flush()
fmt.Printf("The underlying bytes.Buffer writer contains %q\n", uw.Bytes())
}
(Also here)
Output:
readStuff: "source"
The underlying bytes.Buffer writer contains "written"
This way test can consume any io.ReadWriter, not only a specific one. Which is a hint towards your question about go "philosophy".