Runtime access to symbols in Go - go

In writing a Web server in Go, I'd like to be able to dereference symbols at runtime, to allow me to figure out which functions to call from a configuration file, something like the call to the fictional "eval" function in the example below. That would allow me to select handlers from a library of handlers, and to deploy a new server with just a config file. Is there any way to accomplish this in Go?
config.json
{ "url": "/api/apple", "handler": "Apple", "method": "get" }
{ "url": "/api/banana", "handler": "Banana", "method": "get" }
play.go
package main
import (
"github.com/gorilla/mux"
"net/http"
"encoding/json"
"log"
)
type ConfigEntry struct {
URL string `json:"url"`
Method string `json:"method"`
Handler string `json:"handler"`
}
func main() {
ifp, err := os.Open("config.json")
if err != nil {
log.Fatal(err)
}
dec := json.NewDecoder(ifp)
r := mux.NewRouter()
for {
var config ConfigEntry
if err = dec.Decode(&m); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
r.HandleFunc(config.URL, eval(config.Handler + "Handler")).Methods(config.Method)
}
http.Handle("/", r)
http.ListenAndServe(8080, nil)
}
func AppleHandler(w http.ResponseWriter, r *http.Request) (status int, err error) {
w.Write("Apples!\n")
return http.StatusOK, nil
}
func BananaHandler(w http.ResponseWriter, r *http.Request) (status int, err error) {
w.Write("Bananas!\n")
return http.StatusOK, nil
}

There's nothing like eval in Go, which is a good thing since things like that are very dangerous.
What you can do is have a map mapping the handler strings in your config file to the handler functions in your code:
var handlers = map[string]func(http.ResponseWriter, *http.Request) (int, error){
"Apple": AppleHandler,
"Banana": BananaHandler,
}
Then you can register those handlers by simply doing:
handler, ok := handlers[config.Handler]
if !ok {
log.Fatal(fmt.Errorf("Handler not found"))
}
r.HandleFunc(config.URL, handler).Methods(config.Method)

There's some limited way to access things during runtime with the reflect package. However it doesn't allow you to search for all suitable standalone functions in a package. It would be possible if they are all methods on a known struct type/value.
As an alternative your given example you could simply use a map[string]func(...) to store all handlers, initialize it at startup (during init()) and fetch the handlers from there. But that also more or less what the existing http muxes are doing.

Related

Getting same file downloaded multiple times concurrently

I am concurrently downloading files (with a WaitGroup) from a slice of config objects (where each config object contains the URL that needs to be downloaded), but when I use concurrency, I get the same exact data written with every execution.
I believe I included everything below for a minimal reproducible example.
Here are my imports:
package main
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"os"
"path"
"path/filepath"
"strconv"
"strings"
"sync"
)
The method that's looping through my objects and executing the go routine to download each file is here:
func downloadAllFiles(configs []Config) {
var wg sync.WaitGroup
for i, config := range configs {
wg.Add(1)
go config.downloadFile(&wg)
}
wg.Wait()
}
Basically, my function is downloading a file from a URL into a directory stored on NFS.
Here is the download function:
func (config *Config) downloadFile(wg *sync.WaitGroup) {
resp, _ := http.Get(config.ArtifactPathOrUrl)
fmt.Println("Downloading file: " + config.ArtifactPathOrUrl)
fmt.Println(" to location: " + config.getNfsFullFileSystemPath())
defer resp.Body.Close()
nfsDirectoryPath := config.getBaseNFSFileSystemPath()
os.MkdirAll(nfsDirectoryPath, os.ModePerm)
fullFilePath := config.getNfsFullFileSystemPath()
out, err := os.Create(fullFilePath)
if err != nil {
panic(err)
}
defer out.Close()
io.Copy(out, resp.Body)
wg.Done()
}
Here's a minimal part of the Config struct:
type Config struct {
Namespace string `json:"namespace,omitempty"`
Tenant string `json:"tenant,omitempty"`
Name string `json:"name,omitempty"`
ArtifactPathOrUrl string `json:"artifactPathOrUrl,omitempty"`
}
Here are the instance/helper functions:
func (config *Config) getDefaultNfsURLBase() string {
return "http://example.domain.nfs.location.com/"
}
func (config *Config) getDefaultNfsFilesystemBase() string {
return "/data/nfs/location/"
}
func (config *Config) getBaseNFSFileSystemPath() string {
basePath := filepath.Dir(config.getNfsFullFileSystemPath())
return basePath
}
func (config *Config) getNfsFullFileSystemPath() string {
// basePath is like: /data/nfs/location/
trimmedBasePath := strings.TrimSuffix(config.getDefaultNfsFilesystemBase(), "/")
fileName := config.getBaseFileName()
return trimmedBasePath + "/" + config.Tenant + "/" + config.Namespace + "/" + config.Name + "/" + fileName
}
Here is how I'm getting the configs and unmarshalling them:
func getConfigs() string {
b, err := ioutil.ReadFile("pulsarDeploy_example.json")
if err != nil {
fmt.Print(err)
}
str := string(b) // convert content to a 'string'
return str
}
func deserializeJSON(configJson string) []Config {
jsonAsBytes := []byte(configJson)
configs := make([]Config, 0)
err := json.Unmarshal(jsonAsBytes, &configs)
if err != nil {
panic(err)
}
return configs
}
For a minimal example, I think this data for the pulsarDeploy_example.json file should work:
[{ "artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/sample/sample.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName1",
"tenant": "exampleTenant1"
},
{
"artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/sample-calculator/sample-calculator-bundle-2.0.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName2",
"tenant": "exampleTenant1"
},
{
"artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/helloworld/helloworld.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName3",
"tenant": "exampleTenant1"
},
{
"artifactPathOrUrl": "http://www.java2s.com/Code/JarDownload/fabric-activemq/fabric-activemq-demo-7.0.2.fuse-097.jar.zip",
"namespace": "exampleNamespace1",
"name": "exampleName4",
"tenant": "exampleTenant1"
}
]
(Note that the example file URLs were just random Jars I grabbed online.)
When I run my code, instead of downloading each file, it repeatedly downloads the same file, and the information it prints to the console (from the Downloading file: and to location: lines) is exactly the same for each object (instead of printing the messages that are unique to each object), which definitely is a concurrency issue.
This issue reminds me of what happens if you try to run a for loop with a closure and end up locking a single object instance into your loop and executing repeatedly on the same object.
What is causing this behavior, and how do I resolve it?
I'm pretty sure that your guess
This issue reminds me of what happens if you try to run a for loop with a closure and end up locking a single object instance into your loop and executing repeatedly on the same object.
is correct. The simple fix is to "assign to local var" like
for _, config := range configs {
wg.Add(1)
cur := config
go cur.downloadFile(&wg)
}
but I don't like APIs which take waitgroup as a parameter so I suggest
for _, config := range configs {
wg.Add(1)
go func(cur Config) {
defer wg.Done()
cur.downloadFile()
}(config)
}
and change downloadFile signature to func (config *Config) downloadFile() and drop the wg usage in it.

How to use http.ResponseWriter in fasthttp for executing HTML templates etc.?

I've recently moved from golang net/http to fasthttp owing to rave reviews.
As you know, fasthttp doesn't use (w http.ResponseWriter) but only one syntax which is (ctx *fasthttp.RequestCtx).
I tried using ctx.Write but it didn't work.
So, how can I implement http.ResponseWriter in the code below to excute my html templates? Please also give some explanation so that we could all benefit.
Much Thanks for your help!
package main()
import (
"html/template"
"fmt"
"github.com/valyala/fasthttp"
)
type PageData struct {
Title string
}
func init() {
tpl = template.Must(template.ParseGlob("public/templates/*.html"))
}
m := func(ctx *fasthttp.RequestCtx) {
switch string(ctx.Path()) {
case "/":
idx(ctx)
default:
ctx.Error("not found", fasthttp.StatusNotFound)
}
}
fasthttp.ListenAndServe(":8081", m)
}
func idx(ctx *fasthttp.RequestCtx) {
pd := new(PageData)
pd.Title = "Index Page"
err := tpl.ExecuteTemplate(ctx.write, "index.html", pd)
if err != nil {
log.Println("LOGGED", err)
http.Error(ctx.write, "Internal server error", http.StatusInternalServerError)
return
}
}
*fasthttp.RequestCtx implements the io.Writer interface (that's why ctx.Write() exists), which means that you can simply pass ctx as the parameter to ExecuteTemplate():
tpl.ExecuteTemplate(ctx, "index.html", pd)
Also, the http.Error() call will not work, since RequestCtx is not a http.ResponseWriter. Use the RequestCtx's own error function instead:
ctx.Error("Internal server error", http.StatusInternalServerError)

prevent access to files in folder with a golang server

I've a server in golang who handle folder path like that :
fs := http.FileServer(http.Dir("./assets"))
http.Handle("/Images/", fs)
http.ListenAndServe(":8000", nil)
But in this folder there are privates images, and it shouldn't be possible to access files. So how can i secure image access and prevent anybody to access content of folder.
like that for example :
If you want to block a directory using http package, maybe this will be useful to you :
https://groups.google.com/forum/#!topic/golang-nuts/bStLPdIVM6w
package main
import (
"net/http"
"os"
)
type justFilesFilesystem struct {
fs http.FileSystem
}
func (fs justFilesFilesystem) Open(name string) (http.File, error) {
f, err := fs.fs.Open(name)
if err != nil {
return nil, err
}
return neuteredReaddirFile{f}, nil
}
type neuteredReaddirFile struct {
http.File
}
func (f neuteredReaddirFile) Readdir(count int) ([]os.FileInfo, error) {
return nil, nil
}
func main() {
fs := justFilesFilesystem{http.Dir("/tmp/")}
http.ListenAndServe(":8080", http.FileServer(fs))
}
A little wrapper over FileServer() solves your problem, now you have to add some sort of logic to do Authorization, it looks like you have unique names, that's good, so I just filter the image name for you creating a map of names, now you can add something more dynamic like a key/store(memcached, redis. etc.) Hope you can follow the comments
package main
import (
"log"
"net/http"
"strings"
)
// put the allowed hashs or keys here
// you may consider put them in a key/value store
//
var allowedImages = map[string]bool{
"key-abc.jpg": true,
"key-123.jpg": true,
}
func main() {
http.Handle("/Images/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// here we can do any kind of checking, in this case we'll just split the url and
// check if the image name is in the allowedImages map, we can check in a DB or something
//
parts := strings.Split(r.URL.Path, "/")
imgName := parts[len(parts)-1]
if _, contains := allowedImages[imgName]; !contains { // if the map contains the image name
log.Printf("Not found image: %q path: %s\n", imgName, r.URL.Path)
// if the image is not found we write a 404
//
// Bonus: we don't list the directory, so nobody can know what's inside :)
//
http.NotFound(w, r)
return
}
log.Printf("Serving allowed image: %q\n", imgName)
fileServer := http.StripPrefix("/Images/", http.FileServer(http.Dir("./assets")))
fileServer.ServeHTTP(w, r) // StripPrefix() and FileServer() return a Handler that implements ServerHTTP()
}))
http.ListenAndServe(":8000", nil)
}
https://play.golang.org/p/ehrd_AWXim

Go lang Redis PubSub in different go routes for publish and subscribe

Am new to go programming language, and I have a requirement to create Redis PubSub with websocket.
My reference code is here
https://github.com/vortec/orchestrate
Am using following libraries
"golang.org/x/net/websocket"
"github.com/garyburd/redigo/redis"
Everything is working for me like this way, but I don't understand what is "websocket.Handler(handleWSConnection)" here.
I need 2 different go routes for /ws-subscribe and /ws-publish. I don't know anything wrong in this concept?
Doubts
Can I do this way, http.HandleFunc("/ws", handleWSConnection) // Tried this way but am getting "not enough arguments in call to handleWSConnection"
Is there any way to call "handleWSConnection()" as a normal function.
Any suggestions how to write /ws-publish as a different go route
Following is my code
main function
func (wsc *WSConnection) ReadWebSocket() {
for {
var json_data []byte
var message WSMessage
// Receive data from WebSocket
err := websocket.Message.Receive(wsc.socket, &json_data)
if err != nil {
return
}
// Parse JSON data
err = json.Unmarshal(json_data, &message)
if err != nil {
return
}
switch message.Action {
case "SUBSCRIBE":
wsc.subscribe.Subscribe(message.Channel)
case "UNSUBSCRIBE":
wsc.subscribe.Unsubscribe(message.Channel)
case "PUBLISH":
wsc.publish.Conn.Do("PUBLISH", message.Channel, message.Data)
}
}
}
func handleWSConnection(socket *websocket.Conn) {
wsc := &WSConnection {socket: socket}
defer wsc.Uninitialize()
wsc.Initialize()
go wsc.ProxyRedisSubscribe()
wsc.ReadWebSocket()
}
func serveWeb() {
http.Handle("/ws", websocket.Handler(handleWSConnection)) // I need to call this route as funciton
if err := http.ListenAndServe(":9000", nil); err != nil {
log.Fatal("ListenAndServe:", err)
}
}
Done following way, I dont know is it the proper way to do this
http.HandleFunc("/publish", publishHandler)
func publishHandler(conn http.ResponseWriter, req *http.Request) {
log.Println("PUBLISH HANDLER")
wsHandler := websocket.Handler(func(ws *websocket.Conn) {
handleWSConnection(ws)
})
wsHandler.ServeHTTP(conn, req)
}

Golang downcasting list of structs

I want to be able to unmarshal yaml files less rigidly. That is, my library has a predefined number of options the yaml file must have. Then, the user should be able to extend this to include any custom options.
Here is what I have
package main
import (
"net/http"
"yamlcms"
"github.com/julienschmidt/httprouter"
)
type Page struct {
*yamlcms.Page
Title string
Date string
}
func getBlogRoutes() {
pages := []*Page{}
yamlcms.ReadDir("html", pages)
}
// This section is a work in progress, I only include it for loose context
func main() {
router := httprouter.New()
//blogRoutes := getBlogRoutes()
//for _, blogRoute := range *blogRoutes {
// router.Handle(blogRoute.Method, blogRoute.Pattern,
// func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {})
//}
http.ListenAndServe(":8080", router)
}
Here is the yamlcms package:
package yamlcms
import (
"io/ioutil"
"os"
"strings"
"gopkg.in/yaml.v2"
)
type Page struct {
Slug string `yaml:"slug"`
File string `yaml:"file"`
}
func (page *Page) ReadFile(file string) (err error) {
fileContents, err := ioutil.ReadFile(file)
if err != nil {
return
}
err = yaml.Unmarshal(fileContents, &page)
return
}
func isYamlFile(fileInfo os.FileInfo) bool {
return !fileInfo.IsDir() && strings.HasSuffix(fileInfo.Name(), ".yaml")
}
func ReadDir(dir string, pages []*Page) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
pages[i].ReadFile(fileInfo.Name())
}
}
return
}
There is a compiler issue here:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.Page in argument to yamlcms.ReadDir
My main intent in this question is to learn the idiomatic way of doing this kind of thing in Go. Other 3rd-party solutions may exist but I am not immediately interested in them because I have problems like this frequently in Go having to do with inheritance, etc. So along the lines of what I've presented, how can I best (idiomatically) accomplish what I am going for?
EDIT:
So I've made some changes as suggested. Now I have this:
type FileReader interface {
ReadFile(file string) error
}
func ReadDir(dir string, pages []*FileReader) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
(*pages[i]).ReadFile(fileInfo.Name())
}
}
return
}
However, I still get a similar compiler error:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.FileReader in argument to yamlcms.ReadDir
Even though main.Page should be a FileReader because it embeds yamlcms.Page.
EDIT: I forgot that slices of interfaces don't work like that. You'd need to allocate a new slice, convert all pages to FileReaders, call the function, and convert them back.
Another possible solution is refactoring yamlcms.ReadDir to return the contents of the files, so that they could be unmarshaled later:
// In yamlcms.
func ReadYAMLFilesInDir(dir string) ([][]byte, error) { ... }
// In client code.
files := yamlcms.ReadYAMLFilesInDir("dir")
for i := range pages {
if err := yaml.Unmarshal(files[i], &pages[i]); err != nil { return err }
}
The original answer:
There are no such things as inheritance or casting in Go. Prefer composition and interfaces in your designs. In your case, you can redefine your yamlcms.ReadDir to accept an interface, FileReader.
type FileReader interface {
ReadFile(file string) error
}
Both yamlcms.Page and main.Page will implement this, as the latter embeds the former.

Resources