How to share single database connection between multiple packages - go

I have two packages named client and worker. I want to share same ssdb, mysql and redis connection with both the packages.
One more similar problem that i am facing to share auth between these two packages.
app
-> client pkg
-> worker pkg
main.go (contains auth as global variable)
Can anyone please suggest me the best way to implement these two things ?

There's lots of ways to do this and each approach has pros and cons and it really depends on what you are doing. One easy way to do this is to have a third package with the DB connection and import it from the other two packages and from the main package.
app
-> client pkg // import "app/global"
-> worker pkg // import "app/global"
-> global pkg // Contains ssdb and auth as global variables
main.go
Another approach that might be better depending on what you are doing is to have the functions on the client and worker packages accept a db connection as a parameter and from main.go initialize the db and pass it as a parameter when you call a function that needs it.
It depends on what you are doing but for big projects it's easier to maintain if you just have one package doing all your db operations and you access it from all the places you need to do something. That way you don't have to worry about this issue because only one package has to worry about the db connection even if several packages use it.
Edit:
The problem with global variables is that they can be modified at the same time from everywhere in your project and it can introduce race conditions, but there is nothing wrong in using them when this is not an issue.
In this case, you are just setting the value once, when you connect to the DB and then just use the object.
You mentioned you want to have another package for the auth, I recommend just having one package and having in it everything you need to access from more than one package, in this case ssdb and auth.

Here's one approach that is not always obvious to new Go developers, is a little elbow grease to implement but not terribly
complex, and usually works fine in beginner apps:
app
client // imports storage
worker // imports storage
config // all environment-related config goes here
storage // storage-engine-generic interface to the packages below it
ssdb // all ssdb-specific code here
mysql // all mysql-specific code here
redis // ditto
It uses package variables. If you're paranoid about an accidental write to an exported package variable, you can avoid the problem by using unexported package variables. Take advantage of the
limited definition of Exported Identifiers in Go (see language
specification).
In main, call
config.Init(configfile)
storage.Init()
Define your config.Init function to read the config file and set package variables to the connection information for your
databases. If you're using enexported package variables, then allow public read-only access through exported functions. Otherwise you may be able to skip the functions, depending on what other features you want.
In storage, your Init function calls
ssdb.Init()
mysql.Init()
redis.Init()
Then also in storage you'll have public functions that client and server use that aren't specific to a storage engine, such as
func GetImage(id string) ([]byte) {
return mysql.GetImage(id)
}
or whatever is appropriate for your application. The storage level of abstraction may or may not be worth it for you depending on how you change your app in the future. You decide whether it's worth investing in it.
In mysql package, you import config, and you have something like
var db *sql.DB
func Init() {
getDb()
}
func getDb() (*sql.DB) {
if db == nil { // or something
config.Log.Println("Opening db connection to mysql")
db, err := sql.Open("mysql", config.MysqlConnectionString())
// do something with err, possibly have a retry loop
}
return db
}
func GetImage(id string) ([]byte)
db := getDb()
// ...
The functions in the mysql package can use the db unexported package variable, but other packages cannot.
Using an unexported package variable with exported-function read-only access is not a terrible practice or particularly complex. That said, it's usually unecessary. If db were the exported package variable Db, would you suddenly type
mysql.Db, _ = sql.Open("mysql", "LEEERRROOYYYYY!!!!")
in your client code (and also decide to import mysql and sql to do it) and then deploy to production? Why would you be more likely to do that than to intentionally break any other part of your code?
Note that if you just typed
mysql.Db = "LEEERRROOYYYYYY!!!!"
Your application would fail to compile because of a type mismatch.

Related

Handling package and function names dynamically

I'm on my first Golang project, which consists of a small router for an MVC structure. Basically, what I hope it does is take the request URL, separate it into chunks, and forward the execution flow to the package and function specified in those chunks, providing some kind of fallback when there is no match for these values in the application .
An alternative would be to map all packages and functions into variables, then look for a match in the contents of those variables, but this is not a dynamic solution.
The alternative I have used in other languages (PHP and JS) is to reference those names sintatically, in which the language somehow considers the value of the variable instead of considering its literal. Something like: {packageName}.{functionName}() . The problem is that I still haven't found any syntactical way to do this in Golang.
Any suggestions?
func ParseUrl(request *http.Request) {
//out of this URL http://www.example.com/controller/method/key=2&anotherKey=3
var requestedFullURI = request.URL.RequestURI() // returns '/controller/method?key=2key=2&anotherKey=3'
controlFlowString, _ := url.Parse(requestedFullURI)
substrings := strings.Split(controlFlowString.Path, "/") // returns ["controller","method"]
if len(substrings[1]) > 0 {
// Here we'll check if substrings[1] mathes an existing package(controller) name
// and provide some error return in case it does not
if len(substrings[2]) > 0 {
// check if substrings[2] mathes an existing function name(method) inside
// the requested package and run it, passing on the control flow
}else{
// there's no requested method, we'll just run some fallback
}
} else {
err := errors.New("You have not determined a valid controller.")
fmt.Println(err)
}
}
You can still solve this in half dynamic manner. Define your handlers as methods of empty struct and register just that struct. This will greatly reduce amount of registrations you have to do and your code will be more explicit and readable. For example:
handler.register(MyStruct{}) // the implementation is for another question
Following code shows all that's needed to make all methods of MyStruct accessible by name. Now with some effort and help of reflect package you can support the routing like MyStruct/SomeMethod. You can even define struct with some fields witch can serve as branches so even MaStruct/NestedStruct/SomeMethod is possible to do.
dont do this please
Your idea may sound like a good one but believe me its not. Its lot better to use framework like go-chi that is more flexible and readable then doing some reflect madness that no one will understand. Not to mention that traversing type trees in go was newer the fast task. Your routes should not be defined by names of structures in your backend. When you commit into this you will end up with strangely named routes that use PascalCase instead of something-like-this.
What you're describing is very typical of PHP and JavaScript, and completely inappropriate to Go. PHP and JavaScript are dynamic, interpreted languages. Go is a static, compiled language. Rather than trying to apply idioms which do not fit, I'd recommend looking for ways to achieve the same goals using implementations more suitable to the language at hand.
In this case, I think the closest you get to what you're describing while still maintaining reasonable code would be to use a handler registry as you described, but register to it automatically in package init() functions. Each init function will be called once, at startup, giving the package an opportunity to initialize variables and register things like handlers and drivers. When you see things like database driver packages that need to be imported even though they're not referenced, init functions are why: importing the package gives it the chance to register the driver. The expvar package even does this to register an HTTP handler.
You can do the same thing with your handlers, giving each package an init function that registers the handler(s) for that package along with their routes. While this isn't "dynamic", being dynamic has zero value here - the code can't change after it's compiled, which means that all you get from being dynamic is slower execution. If the "dynamic" routes change, you'd have to recompile and restart anyway.

Using interfaces for using methods in a different package

I'm trying to understand the way interfaces can be used.
Below is a contrived example to demonstrate my question.
I have the main package, which instantiates a test database, and then passes this to a test server, where the server is then initialised.
Then there is a call to the server, which does a dummy database insert (using the dummy database dependency, passed on server initialisation).
main.go
package main
import (
"interfaces/database"
"interfaces/server"
)
func main() {
db := database.Start()
s := server.Start(db)
s.HandleInsert()
}
database.go
package database
import "fmt"
type Database struct {
pool string
}
func Start() *Database{
database := &Database{}
database.pool = "examplepool"
return database
}
func(db *Database) Select() {
fmt.Println("Running a Select")
}
func(db *Database) Insert() {
fmt.Println("Running an Insert")
}
func (db *Database) Delete() {
fmt.Println("Running a Delete")
}
server.go
package server
import "fmt"
type Database interface {
Select()
Insert()
Delete()
}
type Server struct {
server string
db Database
}
func Start(db Database) *Server {
fmt.Println("Created Server")
s := &Server{"exampleserver", db}
return s
}
func(s *Server) HandleInsert() {
s.db.Insert()
}
The thing is, in the server package, to make use of the database package, I've had to write out all the methods that the database object has. I've only got three methods, but my Database object could easily have more. This goes against Go's philosophy of having small interfaces. What am I missing here? I don't want to import the Database package into the Server package, as I want to encapsulate each package as much as possible.
Another question is, say I have other packages that want to make use of this Database package. Should they also contain a similar Database interface? Should I maybe have a package called "interfaces" which contains the Database interface, that can then be imported?
The idea for the layout of this code came from this video: https://youtu.be/rWBSMsLG8po
At first glance, I'd say you're doing things in the idiomatic golang way:
Return the type (database Start returns *Database, as it should)
The package defines interfaces for its dependencies (as you're doing)
The size of the interface isn't determined by what a given type exports (i.e. what methods your database.Database type implements), but rather what functionality you need. If your server package only ever needs to use Select, Insert, and Delete, then that's what the interface should be. The type you're passing to the server package could implement SelectNASASecretLizardFiles, but if you're not using it, the server package doesn't have to know the method exists. The server.Database interface remains as simple as it is now.
That's essentially what a small interface means. Golang interfaces are implemented implicitly (sometimes people call it ducktype interfaces). Any type that implements the 3 methods you defined in server.Database can be used as a dependency. This makes your packages really easy to unit-test (mocking is trivial).
The "downside" can be that, if you have several packages depending on the Database type, you can end up with duplicate definitions of the same interface. However, if one of the packages requires access to an additional function (or doesn't need to use the Insert method), changing the interface for that package doesn't affect any of the other packages. This fits with the whole concept of golang packages being self-contained.
In your particular case, though, I think there's room for a judgement call. If you're interfacing with a DB of sorts, I think it's a fair assumption to make that most, if not all, packages will all need to be able to select data. It's common to see a small, base interface defined in a common package:
|
|-- server
| |
| |--> dependencies.go (defines Databse interface for server pkg)
|
|-- foo
| |
| |--> dependencies.go (defines Database interface for this package)
|
|-- common (bad package name, but self explanatory)
| |
| |--> database.go (defines common subset of database interfaces)
Where the interfaces look like this:
package common
type DB interface {
// don't return a slice of maps, this is just an example
Select(query string, args ...interface{}) (rows []map[string]interface{}, err error)
Close() error
}
package server
import "your.project/common"
type Database interface {
common.DB // embedded common interface
Insert(query string, vals ...interface{}) error
Delete(query, id string) error
}
This is a common way to structure your code, while ensuring easy mocking and testing.
Speaking of mocking/testing, just a tip, but have a look at a tool called mockgen. You can have mocks for unit tests generated for your interfaces per-package by adding a single comment like this:
package server
import "your.project/common"
//go:generate go run github.com/golang/mock/mockgen -destination mocks/db_mock.go -package mocks your.project/server Database
type Database interface {
common.DB // embedded common interface
Insert(query string, vals ...interface{}) error
Delete(query, id string) error
}
Running go generate will spit out the mocks you can then import in your unit tests.
Other comments
Something I couldn't help notice is that your database package declares a type called Database. Why is the type exported? and why does it have the same name as the package? Using a type called database.Database is just code smell. Stuttering names should be avoided. Perhaps calling the handle Handle or Conn makes more sense: db.Handle or db.Conn is much more descriptive of what you're actually dealing with, and it's shorter to type.
The function to get the DB connection is also weirdly named (Start). It's a constructor function, so I think it'd make more sense to call it New, resulting in the code:
db := database.New()

Does golang foment no file structure?

I've been looking into golang in order to build a web app, I like the language and everything, but I'm having trouble wrapping my head around the concept of structure in golang. It seems it pretty much forces me to have no file structure, no folders, no division, no separation of concerns. Is there any way to organize the .go files in a way that I'm not seeing? So far file structuring has been a headache and it's the only bad experience I've had with the language. Thank you!
You are partially right. Go does not enforce anything regarding file and package structure, except that it forbids circular dependencies. IMHO, this is a good thing, since you have freedom to choose what best suites you.
However, it puts burden on you to decide what is the best. I have tried few approaches and depending on what I am doing (e.g. library, command line tool, service) I believe different approaches are best.
If you are creating only command line tool, let root package (root of your repository) be main. If it is small tool, that is all you need. It might happen that you command line tool grows, so you might want to separate some stuff to their own that can, but does not have to be, in same repository.
If you are creating library, do the same, except that package name will be name of your library, not main.
If you need combination (something that is useful both as the library and command line tool), I would go with putting library code (everything public for the library) in VCS root, with potential sub-packages and cmd/toolname for your binary.
When it comes to web services, I found it is most practical to follow these guidelines. It is best to read entire blog post, but in short - define your domain in VCS root, create cmd/app (or multiple) as command line entry point and create one package per dependency (e.g. memcache, database, http, etc). Your sub-packages never depend on each other explicitly, they only share domain definitions from root. It takes some getting used to and I am still adapting it to my use case, but so far it looks promising.
As #del-boy said it depends on what you want to do, I went over this problem multiple times but what suited me more when developing a golang web app is to divide your packages by dependencies
- myproject
-- cmd
--- main.go
-- http
--- http.go
-- postgres
--- postgres.go
-- mongodb
--- mongodb.go
myproject.go
myproject.go will contain the Interfaces and Structs that contain the main domain or business models
For example you can have inside myproject.go
type User struct {
MongoID bson.ObjectId `bson:"_id,omitempty"`
PostgresID string
Username string
}
and an Interface like this
type UserService interface {
GetUser(username string) (*User, error)
}
Now in your http package you will handle exposing your api endpoints
//Handler represents an HTTP API interface for our app.
type Handler struct {
Router *chi.Mux // you can use whatever router you like
UserService myproject.UserService
}
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *Request){
//this just a wrapper for the Router ServeHTTP
h.Router.ServeHTTP(w,r)
}
func (h *Handler) someHandler(w http.ResponseWriter, r *Request){
//get the username from the request
//
user := h.UserService.GetUser(username)
}
in your postgres.go you can have a struct that implements UserService
type PostgresUserService struct {
DB *sql.DB
}
and then you implement the service
func (s *PostgresUserService) GetUser(username string) {
//implement the method
}
and the same thing can be done with mongodb
type MongoUserService struct {
Session *mgo.Session
}
func (s *MongoUserService) GetUser(username string) {
//implement the method
}
Now in your cmd/main.go you can have something like this
func main(){
postgresDB, err := postgres.Connect()
mongoSession, err := mongo.Connect()
postgresService := postgres.PostgresUserService{DB: postgresDB}
mongoService := mongo.MongoUserService{Session: mongoSession}
//then pass your services to your http handler
// based on the underlying service your api will act based on the underlying service you passed
myHandler := http.Handler{}
myHandler.UserService = postgresService
}
assuming you changed your underlying store you only have to change it in here and you will not change anything
This design is heavily inspired from this blog, I hope you find it helpful

Should I avoid package singletons in golang?

at the moment I have a package store with following content:
package store
var (
db *Database
)
func Open(url string) error {
// open db connection
}
func FindAll(model interface{}) error {
// return all entries
}
func Close() {
// close db connection
}
This allows me to use store.FindAll from other packages after I have done store.Open in main.go.
However as I saw so far most packages prefer to provide a struct you need to initialize yourself. There are only few cases where this global approach is used.
What are downsides of this approach and should I avoid it?
You can't instantiate connections to 2 storages at once.
You can't easily mock out storage in unit tests of dependent code using convenient tools like gomock.
The standard http package has a ServerMux for generic usecases and but also has one default instance of ServerMux called DefaultServerMux (http://golang.org/pkg/net/http/#pkg-variables) for convenience. So that when you call http.HandleFunc it creates the handler on the default mux. You can find the same approach used in log and many other packages. This is essentially your "singleton" approach.
However, I don't think it's a good idea to follow that pattern in your use case, since the users need to call Open regardless of the default database. And, because of that, using a default instance would not really help and instead would actually make it less convenient:
d := store.Open(...)
defer d.Close()
d.FindAll(...)
is much easier to both write and read than:
store.Open(...)
defer store.Close()
store.FindAll(...)
And, also there are semantic problems: what should happen if someone calls Open twice:
store.Open(...)
defer store.Close()
...
store.Open(...)
store.FindAll(...) // Which db is this referring to?

Cyclic dependencies and interfaces

I am a long time python developer. I was trying out Go, converting an existing python app to Go. It is modular and works really well for me.
Upon creating the same structure in Go, I seem to land in cyclic import errors, a lot more than I want to. Never had any import problems in python. I never even had to use import aliases. So I may have had some cyclic imports which were not evident in python. I actually find that strange.
Anyways, I am lost, trying to fix these in Go. I have read that interfaces can be used to avoid cyclic dependencies. But I don't understand how. I didn't find any examples on this either. Can somebody help me on this?
The current python application structure is as follows:
/main.py
/settings/routes.py contains main routes depends on app1/routes.py, app2/routes.py etc
/settings/database.py function like connect() which opens db session
/settings/constants.py general constants
/apps/app1/views.py url handler functions
/apps/app1/models.py app specific database functions depends on settings/database.py
/apps/app1/routes.py app specific routes
/apps/app2/views.py url handler functions
/apps/app2/models.py app specific database functions depends on settings/database.py
/apps/app2/routes.py app specific routes
settings/database.py has generic functions like connect() which opens a db session. So an app in the apps package calls database.connect() and a db session is opened.
The same is the case with settings/routes.py it has functions that allow apps to add their sub-routes to the main route object.
The settings package is more about functions than data/constants. This contains code that is used by apps in the apps package, that would otherwise have to be duplicated in all the apps. So if I need to change the router class, for instance, I just have to change settings/router.py and the apps will continue to work with no modifications.
There're two high-level pieces to this: figuring out which code goes in which package, and tweaking your APIs to reduce the need for packages to take on as many dependencies.
On designing APIs that avoid the need for some imports:
Write config functions for hooking packages up to each other at run time rather than compile time. Instead of routes importing all the packages that define routes, it can export routes.Register, which main (or code in each app) can call. In general, configuration info probably flows through main or a dedicated package; scattering it around too much can make it hard to manage.
Pass around basic types and interface values. If you're depending on a package for just a type name, maybe you can avoid that. Maybe some code handling a []Page can get instead use a []string of filenames or a []int of IDs or some more general interface (sql.Rows) instead.
Consider having 'schema' packages with just pure data types and interfaces, so User is separate from code that might load users from the database. It doesn't have to depend on much (maybe on anything), so you can include it from anywhere. Ben Johnson gave a lightning talk at GopherCon 2016 suggesting that and organizing packages by dependencies.
On organizing code into packages:
As a rule, split a package up when each piece could be useful on its own. If two pieces of functionality are really intimately related, you don't have to split them into packages at all; you can organize with multiple files or types instead. Big packages can be OK; Go's net/http is one, for instance.
Break up grab-bag packages (utils, tools) by topic or dependency. Otherwise you can end up importing a huge utils package (and taking on all its dependencies) for one or two pieces of functionality (that wouldn't have so many dependencies if separated out).
Consider pushing reusable code 'down' into lower-level packages untangled from your particular use case. If you have a package page containing both logic for your content management system and all-purpose HTML-manipulation code, consider moving the HTML stuff "down" to a package html so you can use it without importing unrelated content management stuff.
Here, I'd rearrange things so the router doesn't need to include the routes: instead, each app package calls a router.Register() method. This is what the Gorilla web toolkit's mux package does. Your routes, database, and constants packages sound like low-level pieces that should be imported by your app code and not import it.
Generally, try to build your app in layers. Your higher-layer, use-case-specific app code should import lower-layer, more fundamental tools, and never the other way around. Here are some more thoughts:
Packages are good for separating independently usable bits of functionality from the caller's perspective. For your internal code organization, you can easily shuffle code between source files in the package. The initial namespace for symbols you define in x/foo.go or x/bar.go is just package x, and it's not that hard to split/join files as needed, especially with the help of a utility like goimports.
The standard library's net/http is about 7k lines (counting comments/blanks but not tests). Internally, it's split into many smaller files and types. But it's one package, I think 'cause there was no reason users would want, say, just cookie handling on its own. On the other hand, net and net/url are separate because they have uses outside HTTP.
It's great if you can push "down" utilities into libraries that are independent and feel like their own polished products, or cleanly layer your application itself (e.g., UI sits atop an API sits atop some core libraries and data models). Likewise "horizontal" separation may help you hold the app in your head (e.g., the UI layer breaks up into user account management, the application core, and administrative tools, or something finer-grained than that). But, the core point is, you're free to split or not as works for you.
Set up APIs to configure behavior at run-time so you don't have to import it at compile time. So, for example, your URL router can expose a Register method instead of importing appA, appB, etc. and reading a var Routes from each. You could make a myapp/routes package that imports router and all your views and calls router.Register. The fundamental idea is that the router is all-purpose code that needn't import your application's views.
Some ways to put together config APIs:
Pass app behavior via interfaces or funcs: http can be passed custom implementations of Handler (of course) but also CookieJar or File. text/template and html/template can accept functions to be accessible from templates (in a FuncMap).
Export shortcut functions from your package if appropriate: In http, callers can either make and separately configure some http.Server objects, or call http.ListenAndServe(...) that uses a global Server. That gives you a nice design--everything's in an object and callers can create multiple Servers in a process and such--but it also offers a lazy way to configure in the simple single-server case.
If you have to, just duct-tape it: You don't have to limit yourself to super-elegant config systems if you can't fit one to your app: maybe for some stuff a package "myapp/conf" with a global var Conf map[string]interface{} is useful.
But be aware of downsides to global conf. If you want to write reusable libraries, they can't import myapp/conf; they need to accept all the info they need in constructors, etc. Globals also risk hard-wiring in an assumption something will always have a single value app-wide when it eventually won't; maybe today you have a single database config or HTTP server config or such, but someday you don't.
Some more specific ways to move code or change definitions to reduce dependency issues:
Separate fundamental tasks from app-dependent ones. One app I work on in another language has a "utils" module mixing general tasks (e.g., formatting datetimes or working with HTML) with app-specific stuff (that depends on the user schema, etc.). But the users package imports the utils, creating a cycle. If I were porting to Go, I'd move the user-dependent utils "up" out of the utils module, maybe to live with the user code or even above it.
Consider breaking up grab-bag packages. Slightly enlarging on the last point: if two pieces of functionality are independent (that is, things still work if you move some code to another package) and unrelated from the user's perspective, they're candidates to be separated into two packages. Sometimes the bundling is harmless, but other times it leads to extra dependencies, or a less generic package name would just make clearer code. So my utils above might be broken up by topic or dependency (e.g., strutil, dbutil, etc.). If you wind up with lots of packages this way, we've got goimports to help manage them.
Replace import-requiring object types in APIs with basic types and interfaces. Say two entities in your app have a many-to-many relationship like Users and Groups. If they live in different packages (a big 'if'), you can't have both u.Groups() returning a []group.Group and g.Users() returning []user.User because that requires the packages to import each other.
However, you could change one or both of those return, say, a []uint of IDs or a sql.Rows or some other interface you can get to without importing a specific object type. Depending on your use case, types like User and Group might be so intimately related that it's better just to put them in one package, but if you decide they should be distinct, this is a way.
Thanks for the detailed question and followup.
Possible partial, but ugly answer:
Have struggled with the import cyclic dependency problem for a year. For a while, was able to decouple enough so that there wasn't an import cycle. My application uses plugins heavily. At the same time, it uses encode/decode libraries (json and gob). For these, I have custom marshall and unmarshall methods, and equivalent for json.
For these to work, the full type name including the package name must be identical on data structures that are passed to the codecs. The creation of the codecs must be in a package. This package is called from both other packages as well as from plugins.
Everything works as long as the codec package doesn't need to call out to any package calling it, or use the methods or interfaces to the methods. In order to be able to use the types from the package in the plugins, the plugins have to be compiled with the package. Since I don't want to have to include the main program in the builds for the plugins, which would break the point of the plugins, only the codec package is included in both the plugins and the main program. Everything works up until I need to call from the codec package in to the main program, after the main program has called in to the codec package. This will cause an import cycle. To get rid of this, I can put the codec in the main program instead of its own package. But, because the specific datatypes being used in the marshalling/unmarshalling methods must be the same in the main program and the plugins, I would need to compile with the main program package for each of the plugins. Further, because I need to the main program to call out to the plugins I need the interface types for the plugins in the main program. Having never found a way to get this to work, I did think of a possible solution:
First, separate the codec in to a plugin, instead of just a package
Then, load it as the first plugin from the main program.
Create a registration function to exchange interfaces with underlying methods.
All encoders and decoders are created by calls in to this plugin.
The plugin calls back to the main program through the registered interface.
The main program and all the plugins use the same interface type package for this.
However, the datatypes for the actual encoded data are referenced in the main program
with a different name, but same underlying type than in the plugins, otherwise the same import cycle exists. to do this part requires doing an unsafe cast. Wrote
a little function that does a forced cast so that the syntax is clean:
(<cast pointer type*>Cast(<pointer to structure, or interface to pointer to structure>).
The only other issue for the codecs is to make sure that when the data is sent to the encoder, it is cast so that the marshall/unmarshall methods recognize the datatype names. To make that easier, can import both the main program types from one package, and the plugin types from another package since they don't reference each other.
Very complex workaround, but don't see how else to make this work.
Have not tried this yet. May still end up with an import cycle when everything is done.
[more on this]
To avoid the import cycle problem, I use an unsafe type approach using pointers. First, here is a package with a little function Cast() to do the unsafe typecasting, to make the code easier to read:
package ForcedCast
import (
"unsafe"
"reflect"
)
// cast function to do casts with to hide the ugly syntax
// used as the following:
// <var> = (cast type)(cast(input var))
func Cast(i interface{})(unsafe.Pointer) {
return (unsafe.Pointer(reflect.ValueOf(i).Pointer()))
}
Next I use the "interface{}" as the equivalent of a void pointer:
package firstpackage
type realstruct struct {
...
}
var Data realstruct
// setup a function to call in to a loaded plugin
var calledfuncptr func(interface)
func callingfunc() {
pluginpath := path.Join(<pathname>, "calledfuncplugin")
plug, err := plugin.Open(pluginpath)
rFunc, err := plug.Lookup("calledfunc")
calledfuncptr = rFunc.(interface{})
calledfuncptr (&Data)
}
//in a plugin
//plugins don't use packages for the main code, are build with -buildmode=plugin
package main
// identical definition of structure
type realstruct struct {
...
}
var localdataptr *realstruct
func calledfunc(needcast interface{}) {
localdataptr = (*realstruct)(Cast(needcast))
}
For cross type dependencies to any other packages, use the "interface{}" as a void pointer and cast appropriately as needed.
This only works if the underlying type that is pointed to by the interface{} is identical wherever it is cast. To make this easier, I put the types in a separate file. In the calling package, they start with the package name. I then make a copy of the type file, change the package to "package main", and put it in the plugin directory so that the types are built, but not the package name.
There is probably a way to do this for the actual data values, not just pointers, but I haven't gotten that to work right.
One of the things I have done is to cast to an interface instead of a datatype pointer. This allows you to send interfaces to packages using the plugin approach, where there is an import cycle. The interface has a pointer to the datatype, and then you can use it for calling the methods on the datatype from the caller from the package that called in to the plugin.
The reason why this works is that the datatypes are not visible outside of the plugin. That is, if I load to plugins, which are both package main, and the types are defined in the package main for both, but are different types with the same names, the types do not conflict.
However, if I put a common package in to both plugins, that package must be identical and have the exact full pathname for where it was compiled from. To accommodate this, I use a docker container to do my builds so that I can force the pathnames to always be correct for any common containers across my plugins.
I did say this was ugly, but it does work. If there is an import cycle because a type in one package uses a type in another package that then tries to use a type from the first package, the approach is to do a plugin that erases both types with interface{}. You can then make method and function calls back and forth doing the casting on the receiving side as needed.
In summary:
Use interface{} to make void pointers (that is, untyped).
Use the Cast() to force them to a pointer type that matches the underlying pointer. Use the plugin type localization so that types in the package main in separate plugins, and in the main program do not conflict If you use a common package between plugins, the path must be identical for all built plugins and the main program. Use the plug package to load the plugins, and exchange function pointers
For one of my issues I'm actually calling from a package in the main program out to a plugin, just to be able to call back to another package in the main program, avoiding the import cycle between the two packages. I ran in to this problem using the json and gob packages with custom marshaller methods. I use the types that are custom marshalled both in my main program, and in other plugins, while at the same time, I want the plugins to be built independent of the main program. I accomplish this by using a package for json and gob encode/decode custom methods that is included both in the main program and the plugins. However, I needed to be able to call back to the main program from the encoder methods, which gave me the import cycle type conflict. The above solution with another plugin specifically to solve the import cycle works. It does create an extra function call, but I have yet to see any other solution to this.
Hope this helps with this issue.
A shorter answer to your question (using interface), that does not take away the correctness and completeness of the other answers, is this example:
UserService is causing cyclic import, where it should not really be called from AuthorizationService. It's just there to be able to extract the user details, so we can declare only the desired functionality in a separated receiver-side interface UserProvider:
https://github.com/tzvatot/cyclic-import-solving-exaple/commit/bc60d7cfcbd4c3b6540bdb4117ab95c3f2987389
Basically, extracting an interface that contains only the required functionality on the receiver side, and use it instead of declaring a dependency on something external.

Resources