I am looking for a good project organization for a multi-platform project with multiple components written in Go. I am aware of the recommended layout from http://golang.org/doc/code.html, but the layout suggested there does not seem to fit my requirements.
Project components are:
server (written in Go)
client, cross-platform (Go)
library, shared between server and client (Go)
some more clients (iOS, Android)
My requirements are:
All components in a single git repository
Keep components separate (e.g. one directory per component)
Go components can be structured into multiple sub-packages
My current approach:
project/ (this is the repository root)
server/
server.go (package main)
src/
server/
package1/
package1.go
...
client/
client.go (package main)
src/
client/
package2/
package2.go
...
lib/
src/
lib/
lib.go
...
client-ios/
...
client-android/
...
To build, I use a Makefile which
Copies lib/ into both server/ and client/
Builds server/ and client/ separately, setting GOPATH every time to the respective directory.
It works, but feels really klunky and is quite different to the recommended code layout.
Here is the alternative I am considering:
project/ (this is the repository root)
gospace/
src/
server/...
client/...
lib/...
client-ios/
...
client-android/
...
With this layout I have a single GOPATH (gospace/) and don't need the klunky Makefile. However, the components are not separated as neatly as in the first alternative (i.e. via a top level directory).
My question: Which project layout would be best suited for both my requirements and Go conventions and tooling support? Are there better options which I did not see yet?
This is how I organized a similar project :
$GOPATH/src/project-root/
lib.go
lib_test.go
server/
server.go
server_test.go
main/
server.go // package main; import "project-root/server"
client/
client.go
client_test.go
main/
client.go //package main; import "project-root/client"
client-ios/
....
client-android/
....
While mostly having server/server.go and client/client.go as package main should work, it's better to separate it so you could embed the client/server in other projects.
It's been a long time since this question was answered (last comment on OP was Feb. 12, 2014, and the latest answer was Apr. 16, 2014). I wasn't around for the state of Go at that time, only really being introduced to it in 2018/19, yet it looks like a lot has changed.
The biggest change that influences the answer to this post, I believe, is Go Modules.
I have a similar project in the works (Go server, Go CLI, Android app, iOS app). I am currently using go version go1.13.7 darwin/amd64. Here is the structure I came with via Go Modules (very much influenced by this post by Ben Johnson):
directory-outside-GOPATH/
go.mod
go.sum
domaintypes.go
cmd/
api/
main.go
cli/
main.go
postgres/
...(library Go code)...
http/
...(library Go code)...
android/
...(Android app)...
iOS/
...(iOS app)...
...(other Go code).../
...
The root directory sits outside of the GOPATH and includes the necessary go.mod and go.sum files. The domaintypes.go file is to illustrate that the only Go files I leave in the root directory is code that will never import anything else - such as defining your domain types/models/service-interfaces/etc. I personally have multiple: user.go, group.go, etc.
The postgres and http directories are libraries I have written, both of which are layered adapters. Each library directory should be isolated to its domain. e.g. HTTP routes should not directly interact with database connections.
The cmd directory is where all executable Go code should live. I want an API server and a separate CLI so this is where that lives. These files should remain relatively small and call all of the other libraries you wrote.
As for the non-Go code, I simply created separate directories in the root directory for both Android and iOS.
Related
I'm having difficulty understanding how to structure a go project using packages, especially when this project it nested within a repository.
I would like to have the following folder structure within a repository for my go application.
github.com/user/repo
└── client
└── server
└── <create go application "package main" code here>
└── package1
└── <create package 1 code here>
I want to create my go application/module within the server directory within the repository. However I'm not sure what is the correct or 'best practice' to do so.
I'm not sure if the go.mod file should be created within the root directory of the repository or the sub directory server?
Should the module name within my go.mod file be github.com/user/repo or github.com/user/repo/server?
If I have a package called package1 within a sub directory called package1 of the server folder, what would the import path be to import it?
The general best practice is to put your go.mod file at the root of the repo, and have only one module per repo. Then all of the packages within the repo will be in the same module and share the same dependencies.
It is possible to define multiple modules within a single repository, but doing so adds a lot of maintenance overhead: if you have multiple modules, they are treated as completely separate; you have to explicitly manage the dependencies between them, and you have to explicitly tag separate releases for them.
For most projects, that extra overhead comes with very little benefit: mainly it allows you to tag stable releases for the different modules at different times. In my experience, for most projects that extra flexibility is not worth the maintenance cost.
I can not find a way to factor out some code from main.go into a local package when using Go modules (go version >= 1.11) outside of $GOPATH.
I am not importing any external dependencies that need to be included into go.mod, I am just trying to organize locally the source code of this Go module.
The file main.go:
package main
// this import does not work
import "./stuff"
func main() {
stuff.PrintBaz()
}
The file ./stuff/bar.go (pretending to be a local package):
package stuff
import "log"
type Bar struct {
Baz int
}
func PrintBaz() {
baz := Bar{42}
log.Printf("Bar struct: %v", baz)
}
The file go.mod (command go mod init foo):
module foo
go 1.12
When executing go run main.go:
If I import "./stuff", then I see build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
If I import "stuff", then I see build command-line-arguments: cannot load stuff: cannot find module providing package stuff.
If I import stuff "./stuff" with a package alias, then I see again: build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
I can not find a way to make local packages work with go modules.
What's wrong with the code above?
How can I import local packages into other Go code inside a project defined with Go modules (file go.mod)?
Module structure
The most common and easiest approach is:
Use a single go.mod per repository, and
Place the single go.mod file in the repository root, and
Use the repository name as the module path declared in the module line in the go.mod
(If you are using a custom import path such as me.io/mymod rather than using a VCS host based import path, then you would use the custom import path instead of the repository name in your go.mod).
For example, if your repo is github.com/my/repo, then you would place a single go.mod in the repo root, with the first line reading module github.com/my/repo. That can be created by cd'ing to the repo root and running go mod init github.com/my/repo. When working with the repo locally, the repo directory can be located wherever is convenient on your filesystem. (If you do not have a repo, see below).
Following this helps you stay on the happy path with modules, and it avoids multiple subtleties.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
There is much more about multi-module repositories in the "Multi-module Repositories" FAQ section on the modules wiki. Those 6 or so FAQs in that section should be read in their entirety by anyone considering veering off the recommendation above.
Arranging packages within a module
Once you have set up your go.mod, you can arrange your packages in directories however you see fit in directories underneath the directory containing the go.mod, as well as in the directory with the go.mod. Three good articles about how to arrange your code in packages:
https://rakyll.org/style-packages/
https://medium.com/#benbjohnson/standard-package-layout-7cdbc8391fc1#.ds38va3pp
https://www.goinggo.net/2017/02/design-philosophy-on-packaging.html
Those articles are classics that pre-date the introduction of modules, but the philosophies in them still apply to how to arrange your packages within a module.
Importing other packages in the same module
When importing another package with modules, you always use the full path including the module path. This is true even when importing another package in the same module. For example, if a module declared its identity in its go.mod as module github.com/my/repo, and you had this organization:
repo/
├── go.mod <<<<< Note go.mod is located in repo root
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
Then pkg1 would import its peer package as import "github.com/my/repo/pkg2". Note that you cannot use relative import paths like import "../pkg2" or import "./subpkg". (This is part of what OP hit above with import "./stuff").
Modules vs. repositories vs. packages vs. import paths
A Go module is a collection of related Go packages that are versioned together as a single unit. Modules record precise dependency requirements and create reproducible builds.
Summarizing the relationship between repositories, modules, and packages:
A repository contains one or more Go modules (most often exactly one module in the repository root).
Each module contains one or more Go packages.
Each package consists of one or more Go source files that all reside in a single directory.
Each Go source code file:
declares its own package with a package foo statement, which must be consistent for all Go files in a single package directory.
automatically has access to other Go source code in the same package, without explicitly importing its own package.
imports code from another package via an import path supplied in an import statement such as import "github.com/my/repo/pkg1". The import path always starts with the module path of that package, regardless of whether that package is in the same module or a different module.
A single module can be used to build multiple executable binaries, but the func main() {...} entry point for each binary must be in its own package main, with each package main in a distinct directory.
Local-only modules
You can create, build and run modules without ever publishing them or placing them on a VCS server.
A module is defined by a tree of Go source files with a go.mod file in the tree's root directory, so you can pick a directory on your local filesystem to serve as the root of your module and place your go.mod there. Everything described above still applies, except there does not need to be a repo.
For example:
some-directory/
├── go.mod <<<< go.mod at module root declaring 'module example.com/foo'
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
If the module path in the first line of that go.mod reads module example.com/foo, then pkg1 would import its peer package as import "example.com/foo/pkg2".
Of course, you can start with a local-only module without a repo, then later start tracking it in a local repo or publish it to a VCS server.
Even for local-only modules, it is a good idea to give the module a globally unique name, such as go mod init my.org/foo, or go mod init github.com/some/repo as if it will be published someday, even if that repo does not yet exist. (If you are in a hurry, it is possible to use a shorter module name without a dot or leading hostname component, such as go mod init temp or go mod init m, but it won’t be fetchable by the go command, and you run the risk of colliding with a current or future standard library package or otherwise seeing confusing error messages later).
If you have multiple local modules, you can point them at each other via replace directives or via the Go 1.18+ workspace feature, though those are more complex workflows than placing everything for a given project in a single module.
Next steps
If you are new to Go modules, it is worthwhile to go through the official "Create a Go Module" tutorial from the Go project. This will likely save you time overall.
If you are stuck right now with an error message related to modules or import paths, there's a very good chance the error will make much more sense after completing the tutorial.
First you have to choose a name for your project and write it to go.mod file. This name belongs to root directory of the project. Each new package you create must be located inside its own subdirectory and its name should match directory name.
go.mod:
module myprojectname
or (preferred way, see #thepudd's answer for details)
module github.com/myname/myproject
Then import your project's packages like:
import myprojectname/stuff
or
import github.com/myname/myproject/stuff
Files of package stuff should be located inside project's stuff directory. You name these files as you like.
Also it's possible to create deeper project structure. For instance, you decided to separate source code files from other ones (like app configs, docker files, static files, etc...). Let's move stuff directory inside pkg, every go file inside pkg/stuff still have stuff package name. To import stuff package just write:
import myprojectname/pkg/stuff
Nothing stops you from creating more levels in the hierarchy like github.com/myuser/myproject/pkg/db/provider/postgresql, where:
github.com/myuser/myproject - project name.
postgresql - package name.
pkg/db/provider/postgresql - path to the package relative to project's root.
You can read more about go modules here: https://github.com/golang/go/wiki/Modules
Check out this repository to get useful information about various patterns used in project organizing: https://github.com/golang-standards/project-layout If you go inside pkg directory you will find out which open source projects use pkg directory in their structure.
Go now provides modules for dependency management, and I'm a little bit confused as to how I should organize my projects.
In traditional $GOPATH mode, I would organize an application as follows:
├─ cmd/
| └─ myapp/
| └─ main.go
├─ otherstuff/
| └─ file.go
└─ README.md, etc.
This is what I see most projects on GitHub doing.
However, now that we have modules, I'm not sure where to put go.mod. Does it go in the project's root directory? Does it go in cmd/[whatever]/? Should I still be putting main.go in the cmd/[whatever] directory or should it be in the project's root directory now?
From the wiki:
A module is a collection of related Go packages that are versioned together as a single unit. Most often, a single version-control repository corresponds exactly to a single module, but alternatively, a single version-control repository can hold multiple modules.
So putting go.mod right next to .git (or equivalent for some other VCS) is almost always the right thing to do. You would only create multiple modules in a single repository if the code in each module is truly independent from the other modules, so that the version of one module does not effect the other modules in any way.
How to setup project structure to produce library and cli with same name in
single repository?
Suppose my project name is project. I want to make it importable with name
project and have executable binary with name project when installed with
go get. My setup currently is like this:
host.com/project/
project/
main.go
core/
project.go
Then, when installed with:
go get host.com/project/project
it installs project as executable which pulls core as dependency. In
core/project.go file, the package has this:
package project
The problem is it is imported with:
import (
"host.com/project/core"
)
And it exports project as name space not core which violates go's convention.
How can I do this?
This isn't a convention, and you may do however you like, but it's a common practice to have a cmd folder in the project root, and all the executables go under that, each in is own folder.
So a good solution in your case could be simply:
host.com/project/
cmd/
project/
project.go
filea.go
fileb.go
In filea.go and fileb.go the package declaration should be:
package project
In cmd/project/project.go package and import declaration should be:
package main
import "host.com/project"
Of course if your project is more complex, you may create further packages under the project root, and it may have multiple commands (multiple main packages), e.g.:
host.com/project/
cmd/
project/
project.go
prjtool/
prjtool.go
packagex/
x.go
packagey/
y.go
filea.go
fileb.go
An example following this layout is the very popular Go Delve debugger (4.5k stars currently), available at https://github.com/derekparker/delve.
A very complex repository example is the golang.org/x/tools package which is a collection of multiple, essential Go tools, available at https://github.com/golang/tools/. Its cmd folder contains more than 20 subfolders (commands), many of the tools (main packages) even consist of multiple .go files (but each using package main declaration, forming a single package of the executable).
Also check out the following related question: What is a sensible way to layout a Go project
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
Note: this question is related to this one, but two years is a very long time in Go history.
What is the standard way to organize a Go project during development ?
My project is a single package mypack, so I guess I put all the .go files in a mypack directory.
But then, I would like to test it during development so I need at least a file declaring the main package, so that I can do go run trypack.go
How should I organize this ? Do I need to do go install mypack each time I want to try it ?
I would recommend reviewing this page on How to Write Go Code
It documents both how to structure your project in a go build friendly way, and also how to write tests. Tests do not need to be a cmd using the main package. They can simply be TestX named functions as part of each package, and then go test will discover them.
The structure suggested in that link in your question is a bit outdated, now with the release of Go 1. You no longer would need to place a pkg directory under src. The only 3 spec-related directories are the 3 in the root of your GOPATH: bin, pkg, src . Underneath src, you can simply place your project mypack, and underneath that is all of your .go files including the mypack_test.go
go build will then build into the root level pkg and bin.
So your GOPATH might look like this:
~/projects/
bin/
pkg/
src/
mypack/
foo.go
bar.go
mypack_test.go
export GOPATH=$HOME/projects
$ go build mypack
$ go test mypack
Update: as of >= Go 1.11, the Module system is now a standard part of the tooling and the GOPATH concept is close to becoming obsolete.
jdi has the right information concerning the use of GOPATH. I would add that if you intend to have a binary as well you might want to add one additional level to the directories.
~/projects/src/
myproj/
mypack/
lib.go
lib_test.go
...
myapp/
main.go
running go build myproj/mypack will build the mypack package along with it's dependencies
running go build myproj/myapp will build the myapp binary along with it's dependencies which probably includes the mypack library.
I have studied a number of Go projects and there is a fair bit of variation. You can kind of tell who is coming from C and who is coming from Java, as the former dump just about everything in the projects root directory in a main package, and the latter tend to put everything in a src directory. Neither is optimal however. Each have consequences because they affect import paths and how others can reuse them.
To get the best results I have worked out the following approach.
myproj/
main/
mypack.go
mypack.go
Where mypack.go is package mypack and main/mypack.go is (obviously) package main.
If you need additional support files you have two choices. Either keep them all in the root directory, or put private support files in a lib subdirectory. E.g.
myproj/
main/
mypack.go
myextras/
someextra.go
mypack.go
mysupport.go
Or
myproj.org/
lib/
mysupport.go
myextras/
someextra.go
main/
mypack.go
mypage.go
Only put the files in a lib directory if they are not intended to be imported by another project. In other words, if they are private support files. That's the idea behind having lib --to separate public from private interfaces.
Doing things this way will give you a nice import path, myproj.org/mypack to reuse the code in other projects. If you use lib then internal support files will have an import path that is indicative of that, myproj.org/lib/mysupport.
When building the project, use main/mypack, e.g. go build main/mypack. If you have more than one executable you can also separate those under main without having to create separate projects. e.g. main/myfoo/myfoo.go and main/mybar/mybar.go.
I find very useful to understand how to organize code in Golang this chapter http://www.golang-book.com/11 of the book written by Caleb Doxsey
There doesn't seem to be a standard way of organizing Go projects but https://golang.org/doc/code.html specifies a best practice for most projects. jdi's answer is good but if you use github or bitbucket and you have additional libraries as well, you should create the following structure:
~/projects/
bin/
pkg/
src/
github.com/
username/
mypack/
foo.go
bar.go
mypack_test.go
mylib/
utillib.go
utillib_test.go
By doing it this way, you can have a separate repository for mylib that can be used for other projects and can be retrieved by "go get". Your mypack project can import your library using "github.com/username/mylib". For more information:
http://www.alexvictorchan.com/2014/11/06/go-project-structure/
Keep the files in the same directory and use package main in all files.
myproj/
your-program/
main.go
lib.go
Then run:
~/myproj/your-program$ go build && ./your-program
Let's explorer how the go get repository_remote_url command manages the project structure under $GOPATH. If we do a go get github.com/gohugoio/hugo It will clone the repository under
$GOPATH/src/repository_remote/user_name/project_name
$GOPATH/src/github.com/gohugoio/hugo
This is a nice way to create your initial project path. Now let's explorer what are the project types out there and how their inner structures are organized. All golang projects in the community can be categorized under
Libraries (no executable binaries)
Single Project (contains only 1 executable binary)
Tooling Projects (contains multiple executable binaries)
Generally golang project files can be packaged under any design principles such as DDD, POD
Most of the available go projects follows this Package Oriented Design
Package Oriented Design encourage the developer to keeps the implementation only inside it's own packages, other than the /internal package those packages can't can communicate with each other
Libraries
Projects such as database drivers, qt can put under this category.
Some libraries such as color, now follows a flat structure without any other packages.
Most of these library projects manages a package called internal.
/internal package is mainly used to hide the implementation from other projects.
Don't have any executable binaries, so no files that contains the main func.
~/$GOPATH/
bin/
pkg/
src/
repository_remote/
user_name/
project_name/
internal/
other_pkg/
Single Project
Projects such as hugo, etcd has a single main func in root level and.
Target is to generate one single binary
Tooling Projects
Projects such as kubernetes, go-ethereum has multiple main func organized under a package called cmd
cmd/ package manages the number of binaries (tools) that we want to build
~/$GOPATH/
bin/
pkg/
src/
repository_remote/
user_name/
project_name/
cmd/
binary_one/
main.go
binary_two/
main.go
binary_three/
main.go
other_pkg/