Golang project structure in an application versus a package - go

My organization uses Rails to develop its app but I'm attempting to re-write one of our back-end processes in Golang because it's much quicker.
I've structure my application with our company a namespace for my app (example.co), and a subfolder for each of the packages within my app.
Each library that I've included (e.g. sqlx, etc...) also has it's own folder.
src/
github.com/
jmoiron/
(sqlx package files)
example.co
my_app/
(my app package files)
model/
(model package files...)
However looking at other packages like sqlx, it appears they scrap this directory structure entirely and put all their files in the root directory
Is this because I'm writing an application and sqlx is a package that's meant to be included in other applications? Or is it just a difference in preference since there's no real accepted "standard"

I did this too on my first project. I have since learned:
the $GOPATH/bin/ pkg/ src/ layout is constructed by go get and similar commands
you can organize your .go files as a single flat project dir or with subfolders (caveat: all .go files in the same folder must have the same package name)
put other people's code in a /vendor directory inside your project root, if it is code your app needs to work (google this, it's the worst part of go imo)
put your own project under your gopath, symlink to it if you want it more accessible
So I would imagine your code could look something like:
/Users/user2490003/MyGoPath/
▾ src/github.com/user2490003/myproject/
▾ model/
user.go
▾ myapp/
myapp.go
▾ vendor/github.com/jmoiron/sqlx/
sqlx.go
main.go
Import the full package references, like this:
// main.go
package main
import (
github.com/jmoiron/sqlx
github.com/user2490003/myproject/myapp
github.com/user2490003/myproject/model
)

I would recommend to start with a layout that seems logical and works at the present moment and then refactor/restructure as needed when your applications grows and evolves.
Using your company namespace is reasonable - I would consider creating a directory for your app underneath it (such as company.co/my_app) and inside of it, subdirectories for library packages (for example, company.co/my_app/db etc.) as well as the cmd one that would contain directories for the actual executables (package main programs) that you want to produce: cmd/exe1, cmd/exe2 etc. This will allow you to have multiple executables as well as multiple library "subpackages" inside of the my_app which can be included independently with the corresponding import path.
Each library that I've included (e.g. sqlx, etc...) also has it's own folder.
If you are ok with using the latest version of dependencies from Github, you don't have to include the dependencies' code into your repositories but instead install them with go get into the build area. If you want to build from a local copy - and for corporate usage, it may be preferable for stability and audit track - you should put them in a vendor subdirectory, for example company.co/my_app/vendor/github.com/jmoiron/sqlx. This way you have control on when you upgrade to a newer version of the dependencies and assurance that the upstream changes will not break your build or otherwise affect your programs without your knowledge and until you have a chance to do thorough testing.

Related

how to organize GO project packages if it will hosted on different repos (GitHub & SourceForge)

Say I want to create a project and host it on GitHub,I HAVE TO create project struct like this:
src/github.com/user/
myproject/
main.go
util/
fileutil.go
and in the main.go ,I have to write the import as:
import github.com/user/myproject/util/fileutil
AND now I also want to host this project onto SourceForge, should I copy the whole project and modify the path ?? it seems not so good enough. Is there any other way to do that ? What I need is just create my project under src folder, and can be hosted to any repo as I wish, without changing the packages.
If developing an app and not a library
If you don't plan to have other projects "import" your own project (this is, if you are implementing an application and not a library), it might be possible for you to avoid any reference to github.com in your local paths.
Note: this is not the recommended approach, but from my tests it does work, you can compile, run and test your code if you structure it this way.
You can create your project structure as follows:
src/
myproject/
main.go
util/
fileutil.go
Then you can import like this:
import "myproject/util/fileutil"
You can then host the contents of the myproject folder anywhere without changing paths within the project files.
If developing a library
If you are actually developing a library others will be able to import, then it gets more complicated since those projects will actually need a full path to import your project.
You can create a "vanity" import path, like myproject.io/... by using the meta tag like described here:
https://golang.org/cmd/go/#hdr-Remote_import_paths
That way, when the go tool queries your myproject.io/ pages, you should respond with a header like this:
<meta name="go-import" content="myproject.io git https://github.com/user/myproject">
And then change that header you return if you decide to move away from github.
Note that this does not prevent users to import your project directly from github, if you want to avoid that you need to use the technique described in the canonical import path spec:
https://golang.org/doc/go1.4#canonicalimports

Golang package not found [duplicate]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have a go project that is starting to become more complex, and want to lay the filesystem out in such a way to reduce pain.
Are there some good examples out there of what makes sense?
Update May 2013: the official documentation is in the section "Code organization"
Go code must be kept inside a workspace.
A workspace is a directory hierarchy with three directories at its root:
src contains Go source files organized into packages (one package per directory),
pkg contains package objects, and
bin contains executable commands.
The go tool builds source packages and installs the resulting binaries to the pkg and bin directories.
The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages.
bin/
streak # command executable
todo # command executable
pkg/
linux_amd64/
code.google.com/p/goauth2/
oauth.a # package object
github.com/nf/todo/
task.a # package object
src/
code.google.com/p/goauth2/
.hg/ # mercurial repository metadata
oauth/
oauth.go # package source
oauth_test.go # test source
Update July 2014: see "Structuring Applications in Go" from Ben Johnson
That article include tips like:
Separate your binary from your application
combining the main.go file and my application logic in the same package has two consequences:
It makes my application unusable as a library.
I can only have one application binary.
The best way I’ve found to fix this is to simply use a “cmd” directory in my project where each of its subdirectories is an application binary.
camlistore/
cmd/
camget/
main.go
cammount/
main.go
camput/
main.go
camtool/
main.go
Library driven development
Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library.
Sometimes you might want users to interact in multiple ways so you create multiple binaries.
For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version.
You can easily do this by organizing your project like this:
adder/
adder.go
cmd/
adder/
main.go
adder-server/
main.go
Users can install your “adder” application binaries with “go get” using an ellipsis:
$ go get github.com/benbjohnson/adder/...
And voila, your user has “adder” and “adder-server” installed!
Don’t go crazy with subpackages
Usually my project’s types are all very related so it fits better from a usability and API standpoint.
These types can also take advantage of calling unexported between them which keeps the API small and clear.
Group related types and code together in each file. If your types and functions are well organized then I find that files tend to be between 200 and 500 SLOC. This might sound like a lot but I find it easy to navigate. 1000 SLOC is usually my upper limit for a single file.
Organize the most important type at the top of the file and add types in decreasing importance towards the bottom of the file.
Once your application starts getting above 10,000 SLOC you should seriously evaluate whether it can be broken into smaller projects.
Note: that last practice isn't always good:
Sorry I just cant agree with this practice.
Separating type to files helps code management, readability, maintenancability, testability.
It may also ensure single responsibility and the follow of open/closed principle…
The rule for not allowing circular dependency is to force we have a clear structure of the packages.
(Alternative February 2013, regarding src only)
You can find the classic layout illustrated in "GitHub Code Layout":
The app and both libraries live on Github, each in its own repository.
$GOPATH is the root of the project - each of your Github repos will be checked out several folders below $GOPATH.
Your code layout would look like this:
$GOPATH/
src/
github.com/
jmcvetta/
useless/
.git/
useless.go
useless_test.go
README.md
uselessd/
.git/
uselessd.go
uselessd_test.go
README.md
Each folder under src/github.com/jmcvetta/ is the root of a separate git checkout.
That attracted some criticisms though, in this reddit page:
I highly recommend not structuring the repo the way you have, it'll break "go get", which is one of the most useful things about Go.
It's far better to write your code for people who do know Go, since they're most likely to be the ones compiling it.
And for people who don't, they'll at least get a feel for the language.
Put the main package in the root of the repo.
Have the assets in a subdirectory (to keep things neat).
Keep the meat of the code in a subpackage (in case anyone wants to reuse it outside your binary).
Include a setup script in the root of the repo so it's easy to find.
It's still only a two step process to download, build, install, and setup.:
"go get <your repo path>": downloads and installs the go code, with a subdir for the assets
$GOPATH/<your repo path>/setup.sh: distributes the assets to the right place and installs the service
I assume that with 'project' you don't mean a Go package but a software you develop.
Otherwise you can get help here and here.
However it's not so much different to writing packages for Go: Use packages, create a folder for each package and combine those packages in your application.
To build yourself an opinion, you can look at trending Go repositories on github: https://github.com/trending/go. Notable examples are cayley
and zeus.
The most popular scheme is probably to have a main Go file and many modules and submodules in their own directories. In case you have many meta files (doc, license, templates, ...) you may want to put the source code into a subdirectory. That's what I did so far.
There is a recommended approach from the authors of Golang that defines how to layout your code to work best with the go tools and to support source control systems
You should probably also have a look into this repo. It shows a lot of ideas how to structure go applications: https://github.com/golang-standards/project-layout

Implications of not using repo path for my own packages

Suppose I decide to keep all personally developed packages organized
as follows:
$GOPATH/
bin/
pkg/
src/
somepkg1
somepkg2
...
somepkgN
Further, suppose there is a great deal of code reuse among them, so I
decide to keep the whole $GOPATH workspace under the same Git
repository (each package could be a submodule), as opposed to more
traditional scenario where subpackages are less coherent (co-existing
solely because of using go get from the same workspace):
$GOPATH/
bin/
pkg/
src/github.com/<me>/
somepkg1
somepkg2
...
somepkgN
I can see that with the former approach (not using github.com/<me>/
in the package paths), go get would not be able to fetch packages as
they are not "declaring" themselves to be available online. However,
one can easily work around that by using git submodules, so all
packages would be fetched in the first place (note it's a tightly
controlled ecosystem so there will be no name clashes).
Is there any other limitation (besides go get) of not using full
paths for packages?
(I am mostly concerned about limitations arising from certain code
refactoring/analysis tools that exploit the repository path as base path convention that
allows go get to look for the package online.)
For the Go compiler and all elements of the go tool except go get the package import path is an almost opaque string containing the import path. You can lay out your code like you want (the compiler itself happy compiles files from different folders into one package). If you don't need or want your code to be go getable there is no need to use a repo path. The analysis and refactoring tools in golang.org/x/tools work on the opaque import paths (as far as I know) and do not access the network.

Project structure for a tool with multiple UIs

I'm playing with golang and made a tool for password generation. Initially it was intended to be used as a command line tool. I later separated the generation logic into a separate package (still same github repository) and left the main function in the root of the project. Now I also want to add a simple web frontend (nothing fancy), but I don't know how to structure the packages.
Am I supposed to put both the command line entry point as well as the web UI into their own packages in the same project (which leaves the root empty). Or maybe I should move the actual generation library to the root and the UIs in separate packages. I guess the other option is to have the UIs in separate projects on github, but they are only going to be used for this library, so it does not seem like a good idea.
I remember seeing in some projects packages named cmd for example, but never have I encountered one, with multiple front ends. Is there a go(-gettable-)way for doing this?
I agree that there's not much point in making separate projects/repositories if they're only going to be used for this library. I would just have a cmd directory with a subdirectory for each executable you're building.
Something like this:
github.com/user/project
generation
cmd
cmdline
main.go
web
main.go
The main.go files can use the functionality that you've broken out into your "generation" package.
The name of the executables produced by go build will be the name of the parent directory, so cmdline and web in this example (you would want to choose better names).
Note: you don't actually have a package cmdline or web. The files in those directories would all be in [their own separate] package main.

What is a sensible way to layout a Go project [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have a go project that is starting to become more complex, and want to lay the filesystem out in such a way to reduce pain.
Are there some good examples out there of what makes sense?
Update May 2013: the official documentation is in the section "Code organization"
Go code must be kept inside a workspace.
A workspace is a directory hierarchy with three directories at its root:
src contains Go source files organized into packages (one package per directory),
pkg contains package objects, and
bin contains executable commands.
The go tool builds source packages and installs the resulting binaries to the pkg and bin directories.
The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages.
bin/
streak # command executable
todo # command executable
pkg/
linux_amd64/
code.google.com/p/goauth2/
oauth.a # package object
github.com/nf/todo/
task.a # package object
src/
code.google.com/p/goauth2/
.hg/ # mercurial repository metadata
oauth/
oauth.go # package source
oauth_test.go # test source
Update July 2014: see "Structuring Applications in Go" from Ben Johnson
That article include tips like:
Separate your binary from your application
combining the main.go file and my application logic in the same package has two consequences:
It makes my application unusable as a library.
I can only have one application binary.
The best way I’ve found to fix this is to simply use a “cmd” directory in my project where each of its subdirectories is an application binary.
camlistore/
cmd/
camget/
main.go
cammount/
main.go
camput/
main.go
camtool/
main.go
Library driven development
Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library.
Sometimes you might want users to interact in multiple ways so you create multiple binaries.
For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version.
You can easily do this by organizing your project like this:
adder/
adder.go
cmd/
adder/
main.go
adder-server/
main.go
Users can install your “adder” application binaries with “go get” using an ellipsis:
$ go get github.com/benbjohnson/adder/...
And voila, your user has “adder” and “adder-server” installed!
Don’t go crazy with subpackages
Usually my project’s types are all very related so it fits better from a usability and API standpoint.
These types can also take advantage of calling unexported between them which keeps the API small and clear.
Group related types and code together in each file. If your types and functions are well organized then I find that files tend to be between 200 and 500 SLOC. This might sound like a lot but I find it easy to navigate. 1000 SLOC is usually my upper limit for a single file.
Organize the most important type at the top of the file and add types in decreasing importance towards the bottom of the file.
Once your application starts getting above 10,000 SLOC you should seriously evaluate whether it can be broken into smaller projects.
Note: that last practice isn't always good:
Sorry I just cant agree with this practice.
Separating type to files helps code management, readability, maintenancability, testability.
It may also ensure single responsibility and the follow of open/closed principle…
The rule for not allowing circular dependency is to force we have a clear structure of the packages.
(Alternative February 2013, regarding src only)
You can find the classic layout illustrated in "GitHub Code Layout":
The app and both libraries live on Github, each in its own repository.
$GOPATH is the root of the project - each of your Github repos will be checked out several folders below $GOPATH.
Your code layout would look like this:
$GOPATH/
src/
github.com/
jmcvetta/
useless/
.git/
useless.go
useless_test.go
README.md
uselessd/
.git/
uselessd.go
uselessd_test.go
README.md
Each folder under src/github.com/jmcvetta/ is the root of a separate git checkout.
That attracted some criticisms though, in this reddit page:
I highly recommend not structuring the repo the way you have, it'll break "go get", which is one of the most useful things about Go.
It's far better to write your code for people who do know Go, since they're most likely to be the ones compiling it.
And for people who don't, they'll at least get a feel for the language.
Put the main package in the root of the repo.
Have the assets in a subdirectory (to keep things neat).
Keep the meat of the code in a subpackage (in case anyone wants to reuse it outside your binary).
Include a setup script in the root of the repo so it's easy to find.
It's still only a two step process to download, build, install, and setup.:
"go get <your repo path>": downloads and installs the go code, with a subdir for the assets
$GOPATH/<your repo path>/setup.sh: distributes the assets to the right place and installs the service
I assume that with 'project' you don't mean a Go package but a software you develop.
Otherwise you can get help here and here.
However it's not so much different to writing packages for Go: Use packages, create a folder for each package and combine those packages in your application.
To build yourself an opinion, you can look at trending Go repositories on github: https://github.com/trending/go. Notable examples are cayley
and zeus.
The most popular scheme is probably to have a main Go file and many modules and submodules in their own directories. In case you have many meta files (doc, license, templates, ...) you may want to put the source code into a subdirectory. That's what I did so far.
There is a recommended approach from the authors of Golang that defines how to layout your code to work best with the go tools and to support source control systems
You should probably also have a look into this repo. It shows a lot of ideas how to structure go applications: https://github.com/golang-standards/project-layout

Resources