Cycle import issue in gorilla mux routing with multiple sub packages - go

Here is my project structure
--main package
--|--child_package1
--|--child_package2
--|--child_package3
I have all the the routes and method call management for API calls listed in main_package
The router Handler from main_package.go looks like this:
func Handlers(db *sql.DB, customeruploadFile string) *mux.Router {
router := mux.NewRouter()
router.HandleFunc("/api1", child_package1.method )
router.HandleFunc("/api2", child_package2.method)
router.HandleFunc("/api3", child_package3.mehtod)
fileHandler := http.FileServer(http.Dir("./client/compiled"))
router.PathPrefix("/").Handler(http.StripPrefix("/", fileHandler))
return router
}
The problem is when I write test cases for child packages, I need this Handlers method to create test server, for that I need to import main_package in child_packages, then as obvious there is cycle imports happening, as child_packages are imported in main_package.
Can any one please suggest me the best approach to tackle this?

I assume your main_package isn't the main package in Go. I think the child_packages shouldn't be under the main_package as our goal is to decouple each package from one another.
This is the pattern that I'm currently using in my project to avoid dependency conflicts:
project/
├── main_package
│ └── main_package.go
├── brokers
│ └── brokers.go
├── child_package1
│   └── child_package1.go
├── child_package2
│   └── child_package2.go
└── child_package3
   └── child_package3.go
Essentially, each package should never have to deal with anything outside of itself (or at least do so with as little as possible). The broker will be the sole party who "negotiates" between any two packages.
// main_package.go
package main_package
import (
"path/to/sql"
"path/to/mux"
"path/to/brokers"
)
// Never use selectors from packages directly
// but create a `Broker` object for each endpoint
var bk1 = brokers.New("/api1")
var bk2 = brokers.New("/api2")
var bk3 = brokers.New("/api3")
func Handlers(db *sql.DB, customeruploadFile string) *mux.Router {
router := mux.NewRouter()
// each broker has its own `MyHandler` function
router.HandleFunc("/api1", bk1.MyHandler)
router.HandleFunc("/api2", bk2.MyHandler)
router.HandleFunc("/api3", bk3.MyHandler)
fileHandler := http.FileServer(http.Dir("./client/compiled"))
router.PathPrefix("/").Handler(http.StripPrefix("/", fileHandler))
return router
}
The brokers package is the central interface for the communication
// brokers.go
package brokers
import (
"path/to/child_package1"
"path/to/child_package2"
"path/to/child_package3"
"net/http"
)
type Broker interface {
MyHandler(http.ResponseWriter, *http.Request)
}
// Factory function to create a `Broker` instance
func New(uri string) Broker {
if uri == "/api1" {
return Broker( new(child_package1.Delegate) )
} else if uri == "/api2" {
return Broker( new(child_package2.Delegate) )
} else if uri == "/api3" {
return Broker( new(child_package3.Delegate) )
}
return nil
}
Now child_packageX is no long decoupled to any internal dependency, provided
it expose a "representative" or Delegate object to talk to the broker.
// child_package1.go
package child_package1
import "net/http"
type Delegate struct {
// Optional parameters can be carried by the Delegate
// to be used in the created Broker anywhere
}
func (d *Delegate) MyHandler(w http.ResponseWriter, r *http.Request) {
// Maybe return a JSON here
}
Each child can have its own MyHandler that does different things for different api calls, without having to know what endpoints they are serving.
// child_package2
package child_package2
import "net/http"
type Delegate struct {}
func (d *Delegate) MyHandler(w http.ResponseWriter, r *http.Request) {
// Maybe return an XML here
}
The main_package doesn't import all the child_packageX, but just the broker package. You can write a test that imports the broker package instead of the actual packages, or you can even write another broker for testing.
package test
import (
"testing"
"path/to/main_package"
)
func TestMain(*testing.T) {
// test the routing in `main_package`
}
You're no longer testing a functionality of a handler function, but one of an endpoint exposed by a broker. This encourage you to write generic handler functions and focus on the higher level endpoints.
package test
import (
"testing"
"path/to/broker"
)
func TestGetJSONAlright(*testing.T) {
bk1 := brokers.New("/api1")
// test if I get JSON here
}
func TestGetXMLAlright(*testing.T) {
bk1 := brokers.New("/api2")
// test if I get XML here
}
This, in my opinion, is a powerful pattern since you can write more "generic" handlers and just plug them in to the routes you want.

Related

Decoupled project structure in Go and still use variables initialized in main.go for other package and testing?

I am switching to Go from Python/Django. In Django I really liked about its modular Apps project structure design, where each App would have separate bussiness models, routes and views. All the apps would then communicate within the central/project's main routing system and so on.
Django project structure Eg:
- myproject
- myproject
- urls.py
- views.py
...
- planner
- urls.py
- views.py
- models.py
...
I am trying to achieve similar project design in Go project:
- myproject
- cmd
- api
- main.go
- routes.go
- handlers.go
- planner
- routes.go
- handlers.go
- models.go
Excerpt from cmd/api/main.go:
package main
...
db, err := sql.Open("pgx", cfg.db.dsn)
...
srv := &http.Server{
Addr: fmt.Sprintf("localhost:%d", app.config.port),
Handler: app.routes()
}
...
Excerpt from cmd/api/routes.go:
package main
func (app *application) routes() *httprouter.Router {
router := httprouter.New()
planner.Routes(router)
return router
}
Excerpt from cmd/planner/routes.go:
package planner
...
func Routes(router *httprouter.Router) {
router.HandlerFunc(http.MethodPost, "/v1/planner/todos", CreateTodoHandler)
}
Excerpt from cmd/planner/models.go:
package planner
type PlannerModel struct {
DB *sql.DB
}
func (p PlannerModel) InsertTodo(todo *Todo) error {
query := `INSERT INTO todos (title, user_id)
VALUES ($1, $2)
RETURNING id, created_at`
return p.DB.QueryRow(query, todo.Title, todo.UserID).Scan(&todo.ID, &todo.CreatedAt)
}
Now the problem is I need to use the DB connection initialized in cmd/api/main.go file from package main into cmd/planner/handlers.go. Since the variable is from main package I cannot import it into my app's (planner) handler functions.
package planner
func CreateTodoHandler(w http.ResponseWriter, r *http.Request) {
var input struct {
Title string `json:"title"`
UserID int64 `json:"user_id"`
}
err := helpers.ReadJSON(w, r, &input)
...
todo := &Todo{
Title: input.Title,
UserID: input.UserID,
}
...
// How to use/inject DB connection dependency into the PlannerModel?
pm := PlannerModel{
DB: // ????
}
err = pm.InsertTodo(todo)
...
}
I think having a global DB variable solves the problem or a reasonable answer I found was to declare the variable outside in a separate package, and initialize it in main.go. Another approach would be to make the Planner/handlers.go to be of package main and create an application struct in the main package to hold all the models of the project and use it inside the handlers, but I guess this would defeat the decoupled architecture design.
I was wondering what is the preferred way or if there are better ways to go about having similar decoupled project structure like Django, and still use variables initialized in main.go into other packages and do tests?
I had a similar experience when I switched from Python/Django to Go.
The solution to access the db connection in every app is to define structs in each app having a field for db connection, and then create the db connection and all the apps structs in the main.
// planner.go
func (t *Todo) Routes(router *httprouter.Router) {
router.HandlerFunc(http.MethodPost, "/v1/planner/todos", t.CreateTodoHandler)
}
// handlers.go
struct Todo {
DB: *sql.DB
}
func (t *Todo) CreateTodo(w http.ResponseWriter, r *http.Request) {
var input struct {
Title string `json:"title"`
UserID int64 `json:"user_id"`
}
err := helpers.ReadJSON(w, r, &input)
...
todo := &Todo{
Title: input.Title,
UserID: input.UserID,
}
...
pm := PlannerModel{
DB: t.DB
}
err = pm.InsertTodo(todo)
...
}
This would solve your current problem but other problems will arise if you don't have a better design for your application.
I'd recommend reading these two blog posts to better understand designing applications and structuring code in Go.
https://www.gobeyond.dev/standard-package-layout/
https://www.gobeyond.dev/packages-as-layers/
My way is using clean architecture with DI.
Example of repository constructor: https://github.com/zubroide/go-api-boilerplate/blob/master/model/repository/user_repository.go#L16-L19
Example of declaring of repository with db connection: https://github.com/zubroide/go-api-boilerplate/blob/master/dic/app.go#L58-L63
Example of using repository in the service declaration: https://github.com/zubroide/go-api-boilerplate/blob/master/dic/app.go#L65-L70
This looks similar with other DI's, for example wire.
Pluses are:
no problems with cyclic dependencies,
simplifying of services dependencies support.

How to manage cyclic dependencies when have interface and its implementation in different packages

I have my project structure looks like this:
Structure of code:
hypervisor
├── hypervisor.go
├── hyperv
│   └── hyperv.go
└── virtualbox
├── vbox.go
└── vboxprops.go
Source code:
//hypervisor/hypervisor.go
package hypervisor
type Hypervisor interface {
Start(vmName string) error
ListMounts(vmName string) ([]MountPath, error)
//....
}
type MountPath struct {
HostPath string
GuestPath string
}
func detect() (Hypervisor, error) {
return &virtualbox.Virtualbox{}, nil // <<1 HERE
}
// ... other code
And have another (nested) package :
//hypervisor/virtualbox/vbox.go
package virtualbox
type Virtualbox struct {
}
func (*Virtualbox) Start(vmName string) error {
return vboxManage("startvm", vmName, "--type", "headless").Run()
}
func (*Virtualbox) ListMounts(vmName string) ([]hypervisor.MountPath, error) { // <<2 HERE
// ....
}
// ... other code
And as seen, of course, such code leads to import cycle not allowed . because of:
hypervisor pcakge referencing virtualbox.VirtualBox type
virtualbox package referencing hypervisor.MountPath type
I know if I move the struct MounthPath to another package would solve the issue, but I don't think is the correct solution design-wise.
Any suggestion?
One of easiest way I would do is to separate entities into entities package for example (in this case: the Hypervisor and Virtualbox struct are entities or whatever you want to call it).
This is most common design I think, so every struct that inner packages use will not cause cyclic deps.
Example of usage: all time package structs are on top package level. time.Time{}, time.Duration{}, etc. time.Duration does not sit on time/duration package.
Following suggestions from Dave Cheney to define interfaces by the caller will avoid cycle dependencies in most cases. But this will only solve flat data models. In your case, you have nested entities ie., HyperVisor has fucntion which returns MounthPath. We can model this in two ways
Define MouthPath in separate package (like you suggested). In
addition, defining the interface in the virtualbox package will help
in long term to provide alternative implementation for Hypervisor.
Let virtualbox define both Hypervisor and MounthPath as interface. One disadvantage is that the hypervisor implementing package use virtualbox.MouthPath interface to satisfy the interface when passed like below.
//hypervisor/hypervisor.go
package hypervisor
type Hypervisor struct{
someField []virtualbox.MountPath
}
type MountPath struct { // this can be used as virtualbox.MountPath
hostPath string
guestPath string
}
func (m *MountPath) HostPath() string { return m.hostPath }
func (m *MountPath) GuestPath() string { return m.guestPath }
func detect() (Hypervisor, error) {
return &virtualbox.Virtualbox{}, nil // <<1 HERE
}
And have another package (Need not be nested)
//hypervisor/virtualbox/vbox.go
package virtualbox
type Hypervisor interface {
Start(vmName string) error
ListMounts(vmName string) ([]MountPath, error)
//....
}
type MountPath interface {
HostPath() string
GuestPath() string
}
type Virtualbox struct {}
func (*Virtualbox) Start(vmName string) error {
return vboxManage("startvm", vmName, "--type", "headless").Run()
}
func (*Virtualbox) ListMounts(vmName string) ([]MountPath, error) { // <<2 HERE
// ....
}

How golang's init() works. I am confused

I have a init() function defined in "config/config.go"
config.go
package config
import(
log "github.com/sirupsen/logrus"
)
func init() {
log.SetReportCaller(true)
}
I have another go file called auth.go in auth package
package auth
import(
log "github.com/sirupsen/logrus"
)
func auth(username string, pwd string) {
//some auth code
log.Info("Auth success")
}
When log.Info() is called in auth.go the log prints as below
2018-11-09T16:38:27+05:30 auth/auth.go:36 level=info msg="Auth success"
What I am confused here is that, how "log" in auth.go is aware of the settings done in config.go. log.SetReportCaller() is in config.go but even when auth.go is logged it takes settings of log.SetReportCaller() done in config.go
Since log.SetReportCaller() is not set in auth.go expected log should be as below without showing line number of caller method.
2018-11-09T16:38:27+05:30 level=info msg="Auth success"
main.go
package main
import (
"path/to/auth"
log "github.com/sirupsen/logrus"
"net/http"
)
func main() {
r := server.Route()
log.Info("Listening on 8080")
http.Handle("/", r)
log.Fatal(http.ListenAndServe(":8080", nil))
}
auth/router.go
package auth
import (
"github.com/gorilla/mux"
"github.com/rs/cors"
"net/http"
)
func Route() http.Handler {
r := mux.NewRouter()
// UI handlers
r.HandleFunc("/", IndexPageHandler)
r.HandleFunc("/login", LoginHandler).Methods("POST")
handler := cors.Default().Handler(r)
return
}
auth/login.go
package auth
import (
"fmt"
"path/to/config"
log "github.com/sirupsen/logrus"
"net/http"
)
func LoginHandler(w http.ResponseWriter, r *http.Request) {
username := r.FormValue("username")
password := r.FormValue("password")
status, userName, mail, _ := auth(username, password)
//some code goes here
}
Kindly explain how this is happening
Check this diagram to understand how init() work: diagram
Initialization order are as follows,
If a package imports other packages, the imported packages are initialised first.
Package level variables are initialised then.
init function of current package is called next. A package can have multiple init functions (either in a single file or distributed across multiple files) and they are called in the order in which they are presented to the compiler.
You will found an example explaining this here.
I am suspecting in dependency tree, there is a common ancestor file that import both config package and auth package.
Here is official doc on initialization: Package initialization
UPD: As you have added respective codes, let's visualize what is happening here. Look at the picture bellow,
What happening:
Your main package start initialing. But it is has imported auth package. So to complete initialization, auth package must be initialized.
auth package start initializing. But it has imported config package. So to complete initialization, config package must be initialized.
config package complete initialization(log.SetReportCaller(true) is called).
auth package complete initialization.
main package complete initialization.
main() function starts executing...
The behavior you're asking about actually has nothing to do with how init() functions work. SetReportCaller sets a global variable in github.com/sirupsen/logrus, not in your config or auth packages. So it doesn't matter where you call that function or where you call log.Info et al; the setting affects all calls to logrus regardless of call origin.

golang : Custom package and 'undefined'

I've read the doc on creating custom packages, etc but I can't seem to pin down what the problem is.
GOPATH=/Users/lrsmith/GoWorkSpace
|->bin
|->pkg
|->src
|->github.com
|->lrsmith
|-> zaphod
|-> zaphod.go
I've done a 'go get github.com/lrsmith/go-icinga2-api/iapi' and it
dropped it into the same dir as 'zaphod' and created and .a file under pkg.
GOPATH=/Users/lrsmith/GoWorkSpace
|->bin/
|->pkg/
|->..../iapi.a
|->src/
|->github.com/
|->lrsmith/
|-> zaphod/
|-> zaphod.go
|-> go-icinga2-api/
zaphod.go is very simple right now
package main
import (
"github.com/lrsmith/go-icinga2-api/iapi"
)
func main () {
t := iapi.Config("zaphod","beeblebrox","http://localhost",true)
}
When I do a go build in the zaphod directory I get
./zaphod.go:11: undefined: iapi.Config
I've read through the docs, checked cases and tried different structures but I can't seem to get it to load the package and let me call iapi.Config. The iapi code works and if I build something in the go-icinga2-api directory it works fine and the test all pass.
I want to create a separate project/code base that imports the go-icinga2-api and uses it, but can't seem to get it work.
Thanks
Len
Added info. The structure for go-icinga2-api is
go-icinga2-api
|-> iapi
|-> client.go
|-> client_test.go
|-> host.go
.......
client.go is
// Package iapi provides a client for interacting with an Icinga2 Server
package iapi
import (
"bytes"
"crypto/tls"
"encoding/json"
"fmt"
"net/http"
)
// Server ... Use to be ClientConfig
type Server struct {
Username string
Password string
BaseURL string
AllowUnverifiedSSL bool
httpClient *http.Client
}
// func Config ...
func (server *Server) Config(username, password, url string, allowUnverifiedSSL bool) (*Server, error) {
// TODO : Add code to verify parameters
return &Server{username, password, url, allowUnverifiedSSL, nil}, nil
}
I've tried with the .go files up one level, i.e. not nested underneath iapi/ to for the same results.
Updated Answer
In client.go, it looks like you're trying to use Config as a constructor for Server structs. Since Config is defined with a receiver (func (server *Server)), it's a method of Server and can't be called directly. You're code should work if you remove (server *Server).
It's idiomatic to name constructors New[type being returned], or New if the type is the same name as the package.
From the 3rd paragraph of the Package Names section of Idiomatic Go:
the function to make new instances of ring.Ring—which is the definition of a constructor in Go—would normally be called NewRing, but since Ring is the only type exported by the package, and since the package is called ring, it's called just New, which clients of the package see as ring.New
Original Answer
The import path should reference a directory. In your code, you then reference that package by whatever name is used in the package [name] in the .go files in that directory.
For example, if github.com/lrsmith/go-icinga2-api contains a file called api.go with the line package iapi, your importing package should look like this:
package main
import (
"github.com/lrsmith/go-icinga2-api"
)
func main () {
t := iapi.Config("zaphod","beeblebrox","http://localhost",true)
}
Note the declaration of the Config() function:
func (server *Server) Config(username, password, url string, allowUnverifiedSSL bool) (*Server, error)
It's a "method" that should be applied to a Server object:
server.Config(...)
Thus you need to first create a Server object (or you could try with nil):
var server iapi.Server
server, err := server.Config(...)
You're trying to run it as if it had the following declaration:
func Config(username, password, url string, allowUnverifiedSSL bool) (*Server, error)

How to persist "package" states in Go?

My goal is to encapsulate in one module/package.
Main package:
package main
import (
"github.com/zenazn/goji"
"./routes"
)
func main(){
routes.Setup()
goji.Serve()
}
And another package:
package routes
import "github.com/zenazn/goji"
func Setup() {
goji.Get("/static", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, "static!")
})
}
How can I do this?
goji, in your example, is a package. Not a variable.
You cannot pass packages around like this.
If you look at the example on the goji github page
You simply just call goji.Get from your Init function, and goji.Serve from your main
route.go
package route
import "route"
import "github.com/zenazn/goji"
func Init(){
goji.Get("/hello/:name", hello)
}
main.go
package main
import "github.com/zenazn/goji"
func main(){
route.Init()
goji.Serve()
}
Packages in go export constants, variables, types and functions that have uppercase letters as their name. The package itself is not something directly manipulatable by the program.
The package goji should be exporting a variable named something like goji.Goji if you want to directly access it from other packages. A better solution is to provide some functions in the package that allow you to register your functions/helpers.
You could also export a function from goji like:
func Set(s string, func(w http.ResponseWriter, r *http.Request)) { ... }
that could be used by other packages:
goji.Set("/static", myFunc)
The error you had "use of package goji without selector" is saying you can't use the name of the package without specifying which exported value you want from the package. It's expecting goji.something not goji by itself.
The function init() inside go files has special properties: see http://golang.org/ref/spec#Program_initialization_and_execution

Resources