I'm having difficulty understanding how to structure a go project using packages, especially when this project it nested within a repository.
I would like to have the following folder structure within a repository for my go application.
github.com/user/repo
└── client
└── server
└── <create go application "package main" code here>
└── package1
└── <create package 1 code here>
I want to create my go application/module within the server directory within the repository. However I'm not sure what is the correct or 'best practice' to do so.
I'm not sure if the go.mod file should be created within the root directory of the repository or the sub directory server?
Should the module name within my go.mod file be github.com/user/repo or github.com/user/repo/server?
If I have a package called package1 within a sub directory called package1 of the server folder, what would the import path be to import it?
The general best practice is to put your go.mod file at the root of the repo, and have only one module per repo. Then all of the packages within the repo will be in the same module and share the same dependencies.
It is possible to define multiple modules within a single repository, but doing so adds a lot of maintenance overhead: if you have multiple modules, they are treated as completely separate; you have to explicitly manage the dependencies between them, and you have to explicitly tag separate releases for them.
For most projects, that extra overhead comes with very little benefit: mainly it allows you to tag stable releases for the different modules at different times. In my experience, for most projects that extra flexibility is not worth the maintenance cost.
Related
I am developing multiple microservices, which require different modules (which shall be available like modules on github, but private)
My first tests with Go were all located within one package, which gets quite messy after some time
I'm coming from the Java side of programming - with loads of packages - which keep stuff clear and clean.
(This also works with public modules like on github.com/xyz/module1 github.com/xyz/module2 github.com/xyz/module3)
I just need this for private modules - how can I do so?
This is what I tried yet:
Directory of my go sources:
C:\my\path\to\go\src\
In this dir I have multiple subdirs containing modules (actually more than listed here)
my-module1
my-module2
my-module3
For each folder I called go mod init but I get the message that
package my-module1/util is not in GOROOT (c:\go\src\my-module1\util)
which is obviously right, as my private libraries reside in C:\my\path\to\go\src\
Importing packages from github with go get ... is working without troubles (those packages will be loaded but copied to c:\go\src)
Working with all files in one folder works but is not the desired solution (I need to create multiple microservices therefore I want to be able to create different projects with custom executeables and or tests)
What am I doing wrong?
If more information is needed I will provide it - just let me know what
NOTE: packages without go file in package main cannot be installed via go install. This system looks pretty confusing to me - as modules cannot be found...
finally i created a multimodule
means i have a root folder (lets call it root) which contains a go.mod file
this defines the root module
it requires all my submodules
and it replaces the absolute path into a relative.
root
* aa
* ab
* bc
main gets defined as
module root
go 1.15
require (
root/aa v0.0.0
root/ab v0.0.0
root/bc v0.0.0
)
replace (
root/aa => ./aa
root/ab => ./ab
root/bc => ./bc
)
all the submodules have a very simple definition (this is equal for all, as long as you do not have any dependencies)
module root/aa
I can not find a way to factor out some code from main.go into a local package when using Go modules (go version >= 1.11) outside of $GOPATH.
I am not importing any external dependencies that need to be included into go.mod, I am just trying to organize locally the source code of this Go module.
The file main.go:
package main
// this import does not work
import "./stuff"
func main() {
stuff.PrintBaz()
}
The file ./stuff/bar.go (pretending to be a local package):
package stuff
import "log"
type Bar struct {
Baz int
}
func PrintBaz() {
baz := Bar{42}
log.Printf("Bar struct: %v", baz)
}
The file go.mod (command go mod init foo):
module foo
go 1.12
When executing go run main.go:
If I import "./stuff", then I see build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
If I import "stuff", then I see build command-line-arguments: cannot load stuff: cannot find module providing package stuff.
If I import stuff "./stuff" with a package alias, then I see again: build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
I can not find a way to make local packages work with go modules.
What's wrong with the code above?
How can I import local packages into other Go code inside a project defined with Go modules (file go.mod)?
Module structure
The most common and easiest approach is:
Use a single go.mod per repository, and
Place the single go.mod file in the repository root, and
Use the repository name as the module path declared in the module line in the go.mod
(If you are using a custom import path such as me.io/mymod rather than using a VCS host based import path, then you would use the custom import path instead of the repository name in your go.mod).
For example, if your repo is github.com/my/repo, then you would place a single go.mod in the repo root, with the first line reading module github.com/my/repo. That can be created by cd'ing to the repo root and running go mod init github.com/my/repo. When working with the repo locally, the repo directory can be located wherever is convenient on your filesystem. (If you do not have a repo, see below).
Following this helps you stay on the happy path with modules, and it avoids multiple subtleties.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
There is much more about multi-module repositories in the "Multi-module Repositories" FAQ section on the modules wiki. Those 6 or so FAQs in that section should be read in their entirety by anyone considering veering off the recommendation above.
Arranging packages within a module
Once you have set up your go.mod, you can arrange your packages in directories however you see fit in directories underneath the directory containing the go.mod, as well as in the directory with the go.mod. Three good articles about how to arrange your code in packages:
https://rakyll.org/style-packages/
https://medium.com/#benbjohnson/standard-package-layout-7cdbc8391fc1#.ds38va3pp
https://www.goinggo.net/2017/02/design-philosophy-on-packaging.html
Those articles are classics that pre-date the introduction of modules, but the philosophies in them still apply to how to arrange your packages within a module.
Importing other packages in the same module
When importing another package with modules, you always use the full path including the module path. This is true even when importing another package in the same module. For example, if a module declared its identity in its go.mod as module github.com/my/repo, and you had this organization:
repo/
├── go.mod <<<<< Note go.mod is located in repo root
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
Then pkg1 would import its peer package as import "github.com/my/repo/pkg2". Note that you cannot use relative import paths like import "../pkg2" or import "./subpkg". (This is part of what OP hit above with import "./stuff").
Modules vs. repositories vs. packages vs. import paths
A Go module is a collection of related Go packages that are versioned together as a single unit. Modules record precise dependency requirements and create reproducible builds.
Summarizing the relationship between repositories, modules, and packages:
A repository contains one or more Go modules (most often exactly one module in the repository root).
Each module contains one or more Go packages.
Each package consists of one or more Go source files that all reside in a single directory.
Each Go source code file:
declares its own package with a package foo statement, which must be consistent for all Go files in a single package directory.
automatically has access to other Go source code in the same package, without explicitly importing its own package.
imports code from another package via an import path supplied in an import statement such as import "github.com/my/repo/pkg1". The import path always starts with the module path of that package, regardless of whether that package is in the same module or a different module.
A single module can be used to build multiple executable binaries, but the func main() {...} entry point for each binary must be in its own package main, with each package main in a distinct directory.
Local-only modules
You can create, build and run modules without ever publishing them or placing them on a VCS server.
A module is defined by a tree of Go source files with a go.mod file in the tree's root directory, so you can pick a directory on your local filesystem to serve as the root of your module and place your go.mod there. Everything described above still applies, except there does not need to be a repo.
For example:
some-directory/
├── go.mod <<<< go.mod at module root declaring 'module example.com/foo'
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
If the module path in the first line of that go.mod reads module example.com/foo, then pkg1 would import its peer package as import "example.com/foo/pkg2".
Of course, you can start with a local-only module without a repo, then later start tracking it in a local repo or publish it to a VCS server.
Even for local-only modules, it is a good idea to give the module a globally unique name, such as go mod init my.org/foo, or go mod init github.com/some/repo as if it will be published someday, even if that repo does not yet exist. (If you are in a hurry, it is possible to use a shorter module name without a dot or leading hostname component, such as go mod init temp or go mod init m, but it won’t be fetchable by the go command, and you run the risk of colliding with a current or future standard library package or otherwise seeing confusing error messages later).
If you have multiple local modules, you can point them at each other via replace directives or via the Go 1.18+ workspace feature, though those are more complex workflows than placing everything for a given project in a single module.
Next steps
If you are new to Go modules, it is worthwhile to go through the official "Create a Go Module" tutorial from the Go project. This will likely save you time overall.
If you are stuck right now with an error message related to modules or import paths, there's a very good chance the error will make much more sense after completing the tutorial.
First you have to choose a name for your project and write it to go.mod file. This name belongs to root directory of the project. Each new package you create must be located inside its own subdirectory and its name should match directory name.
go.mod:
module myprojectname
or (preferred way, see #thepudd's answer for details)
module github.com/myname/myproject
Then import your project's packages like:
import myprojectname/stuff
or
import github.com/myname/myproject/stuff
Files of package stuff should be located inside project's stuff directory. You name these files as you like.
Also it's possible to create deeper project structure. For instance, you decided to separate source code files from other ones (like app configs, docker files, static files, etc...). Let's move stuff directory inside pkg, every go file inside pkg/stuff still have stuff package name. To import stuff package just write:
import myprojectname/pkg/stuff
Nothing stops you from creating more levels in the hierarchy like github.com/myuser/myproject/pkg/db/provider/postgresql, where:
github.com/myuser/myproject - project name.
postgresql - package name.
pkg/db/provider/postgresql - path to the package relative to project's root.
You can read more about go modules here: https://github.com/golang/go/wiki/Modules
Check out this repository to get useful information about various patterns used in project organizing: https://github.com/golang-standards/project-layout If you go inside pkg directory you will find out which open source projects use pkg directory in their structure.
I have a go package located on my filesystem (not in the $GOPATH), called bitbucket.org/me/awesome.
~/awesome> tree
.
├── main.go
├── go.mod
├── go.sum
├── subpackageA
│ └── main.go
My go.mod looks like:
module bitbucket.org/me/awesome
require (
... # lots of external dependencies
)
replace bitbucket.org/me/awesome => ./
In main.go in my top-level directory, I call a subpackage like follows:
import "bitbucket.org/me/awesome/subpackageA"
which all seems pretty normal. go get works. However, when I clone this entire repository somewhere else (say in a Docker image) and run go get for the first time, I get errors like:
package bitbucket.org/me/awesome/subpackageA: https://api.bitbucket.org/2.0/repositories/me/awesome?fields=scm: 403 Forbidden,
which means it's not using the local filesystem version of the packages, even though I told it to with the replace directive in the go.mod file.
What am I doing wrong? How do I ensure that subpackages are used from the filesystem instead of attempting to be fetched from the internet?
Go has no (real) notion of "subpackage". All packages are basically treated equal. This means that a replace bitbucket.org/me/awesome does not influence package bitbucket.org/me/awesome/subpackageA as these are two individual, unrelated packages. The folder layout does not introduce a relation of subpackageA to awsome, or the other way around *).
So you need to add an individual replace directive for subpackageA
replace bitbucket.org/me/awesome/subpackageA => ./subpackageA
*) Nitpicking for absolute correctness: Folder layout does have influence for folders named internal (cannot be imported from other projects), for folders named vendor (which may contain vendored packages) and searching for a go.mod file stops at the repo root.
For another approach, you can have go.mod like this:
module awesome
Then call subpackage like this:
import "awesome/subpackageA"
https://golang.org/doc/code.html
Go now provides modules for dependency management, and I'm a little bit confused as to how I should organize my projects.
In traditional $GOPATH mode, I would organize an application as follows:
├─ cmd/
| └─ myapp/
| └─ main.go
├─ otherstuff/
| └─ file.go
└─ README.md, etc.
This is what I see most projects on GitHub doing.
However, now that we have modules, I'm not sure where to put go.mod. Does it go in the project's root directory? Does it go in cmd/[whatever]/? Should I still be putting main.go in the cmd/[whatever] directory or should it be in the project's root directory now?
From the wiki:
A module is a collection of related Go packages that are versioned together as a single unit. Most often, a single version-control repository corresponds exactly to a single module, but alternatively, a single version-control repository can hold multiple modules.
So putting go.mod right next to .git (or equivalent for some other VCS) is almost always the right thing to do. You would only create multiple modules in a single repository if the code in each module is truly independent from the other modules, so that the version of one module does not effect the other modules in any way.
I am looking for a good project organization for a multi-platform project with multiple components written in Go. I am aware of the recommended layout from http://golang.org/doc/code.html, but the layout suggested there does not seem to fit my requirements.
Project components are:
server (written in Go)
client, cross-platform (Go)
library, shared between server and client (Go)
some more clients (iOS, Android)
My requirements are:
All components in a single git repository
Keep components separate (e.g. one directory per component)
Go components can be structured into multiple sub-packages
My current approach:
project/ (this is the repository root)
server/
server.go (package main)
src/
server/
package1/
package1.go
...
client/
client.go (package main)
src/
client/
package2/
package2.go
...
lib/
src/
lib/
lib.go
...
client-ios/
...
client-android/
...
To build, I use a Makefile which
Copies lib/ into both server/ and client/
Builds server/ and client/ separately, setting GOPATH every time to the respective directory.
It works, but feels really klunky and is quite different to the recommended code layout.
Here is the alternative I am considering:
project/ (this is the repository root)
gospace/
src/
server/...
client/...
lib/...
client-ios/
...
client-android/
...
With this layout I have a single GOPATH (gospace/) and don't need the klunky Makefile. However, the components are not separated as neatly as in the first alternative (i.e. via a top level directory).
My question: Which project layout would be best suited for both my requirements and Go conventions and tooling support? Are there better options which I did not see yet?
This is how I organized a similar project :
$GOPATH/src/project-root/
lib.go
lib_test.go
server/
server.go
server_test.go
main/
server.go // package main; import "project-root/server"
client/
client.go
client_test.go
main/
client.go //package main; import "project-root/client"
client-ios/
....
client-android/
....
While mostly having server/server.go and client/client.go as package main should work, it's better to separate it so you could embed the client/server in other projects.
It's been a long time since this question was answered (last comment on OP was Feb. 12, 2014, and the latest answer was Apr. 16, 2014). I wasn't around for the state of Go at that time, only really being introduced to it in 2018/19, yet it looks like a lot has changed.
The biggest change that influences the answer to this post, I believe, is Go Modules.
I have a similar project in the works (Go server, Go CLI, Android app, iOS app). I am currently using go version go1.13.7 darwin/amd64. Here is the structure I came with via Go Modules (very much influenced by this post by Ben Johnson):
directory-outside-GOPATH/
go.mod
go.sum
domaintypes.go
cmd/
api/
main.go
cli/
main.go
postgres/
...(library Go code)...
http/
...(library Go code)...
android/
...(Android app)...
iOS/
...(iOS app)...
...(other Go code).../
...
The root directory sits outside of the GOPATH and includes the necessary go.mod and go.sum files. The domaintypes.go file is to illustrate that the only Go files I leave in the root directory is code that will never import anything else - such as defining your domain types/models/service-interfaces/etc. I personally have multiple: user.go, group.go, etc.
The postgres and http directories are libraries I have written, both of which are layered adapters. Each library directory should be isolated to its domain. e.g. HTTP routes should not directly interact with database connections.
The cmd directory is where all executable Go code should live. I want an API server and a separate CLI so this is where that lives. These files should remain relatively small and call all of the other libraries you wrote.
As for the non-Go code, I simply created separate directories in the root directory for both Android and iOS.