I've been playing with Go modules and I was wondering what the best practice is in terms of the following directory structure:
project
├── go.mod
├── main.go
└── players
├── go.mod
├── players.go
└── players_test.go
I was having problems importing the players package into my root project at first, but I noticed I could do this in the root go.mod file
module github.com/<name>/<project>
require (
github.com/<name>/players v0.0.0
)
replace github.com/<name>/players => ./players
This then allows me to do import "github.com/<name>/players" in my main.go file.
Now this approach works and was taken from here but I'm not sure if that's the correct approach for this or whether this approach is just meant for updating a local package temporarily while it's outside version control.
Another option, that seems a little overkill, is to make every module its own repository?
TL;DR; - What's the best practice approach to having multiple modules within the same repository and importing them in in other modules / a root main.go file?
In general a module should be a collection of packages.
But still you can create modules of single packages. As Volker said, this might only make sense, if you want these packages to have a different lifecycle. It could also make sense, when you want to import these modules from another project and don't want the overhead of the whole collection of packages.
In General:
A module is a collection of related Go packages that are versioned together as a single unit.
Modules record precise dependency requirements and create reproducible builds.
Most often, a version control repository contains exactly one module defined in the repository root. (Multiple modules are supported in a single repository, but typically that would result in more work on an on-going basis than a single module per repository).
Summarizing the relationship between repositories, modules, and packages:
A repository contains one or more Go modules.
2. Each module contains one or more Go packages.
3. Each package consists of one or more Go source files in a single directory.
Source of the Quote: https://github.com/golang/go/wiki/Modules#modules
To answer the question:
You can do it the way you have shown in your approach
I understand this is an old question, but there are some more details that are worth mentioning when managing multiple modules in one repository, with or without go.work.
TL;DR
Each approach has pros and cons, but if you are working on a large code base with many modules, I'd suggest sticking to use version handling based on commits or tags, and use Go Workspace for your day to day development.
Go Module Details
replace Directive with No Versioning
When you use replace directive pointing to a local directory, you will find the version of the dependency module as v0.0.0-00010101000000-000000000000. Essentially you get no version information.
With the main go.mod defined with github.com/name/project module path, github.com/name/project module cannot make a reproducible build, because the dependency target for replace directive may have had its content updated. This can be especially problematic if the dependency target of github.com/name/project/players is used by many modules. Any change in such a common package can result in a behaviour change for all the dependents, all at the same time.
If that's not your concern, replace directive should work absolutely fine. In such a setup, go.work may be a layer you don't really need.
With Versioning
If you want to ensure version setup works for reproducible and deterministic build for multiple modules, you can take a few different approaches.
One go.mod, one repository
This is probably the easiest approach. For each module, there is a clear commit history and versioning. As long as you refer to the module via remote repository, this is probably the easiest setup to start with, and dependency setup is very clear.
However, note that this approach would mean you'd need to manage multiple repositories, and making go.work to help is going to require appropriate local directory mapping, which can be difficult for someone new to the code base.
Commit based versioning
It is still possible to deterministically define dependency with version information so that you can build your code, within a single repository. Commit based approach requires least step, and still works nicely. There are some catches to be noted, though.
For github.com/name/project to have a dependency for github.com/name/project/players, you need to ensure the code you need is in the remote repository. This is because github.com/name/project will pull the code and commit information from the remote repository, even if the same code is available on your local copy of the repository. This ensures that the version of github.com/name/project/players is taken from the commit reference, such as v0.1.1-0.20220418015705-5f504416395d (ref: details of "pseudo-version")
The module name must match up the directory structure. For example, if you have the single repository github.com/name/project, and module under /src/mymodule/, the module name must be github.com/name/project/src/mymodule. This is because when module path resolution takes place, Go finds the root of repository (in the above example, this would be github.com/name/project.git), and then tries to follow the directory path based on the module name.
If you are working in a private repository, you will need to ensure go.sum check doesn't block you. You can simply use GOPRIVATE=github.com/name/project to specify paths you don't want the checksum verification to be skipped.
Tag based versioning
Instead of using the commit SHA, you can use Git tags.
But because there could be many modules in one repository, Go Module needs to find which tag maps to which. For example, with the following directory structure:
# All assumed to be using `github.com/name/project` prefix before package name
mypackage/ # v1.0.0
anotherpackage/ # v0.5.1
nested/dependency/ # v0.8.3
You will need to create tags in github.com/name/project, named exactly to match the directory structure, such that:
mypackage/v1.0.0
anotherpackage/v0.5.1
nested/dependency/v0.8.3
This way, each tag is correctly referenced by Go Module, and your dependency can be kept deterministic.
go.work Behaviour
If you have go.work on a parent directory with go work use github.com/name/project/players, etc., that takes precedence and uses the local files. This is even when you have a version specified in your go.mod.
For local development, which spans across multiple projects, Go Workspace is a great way to work on multiple things at once, without needing to push the code change for the dependency only first. But at the same time, actual release will still require broken up commits, so that first commit can be referenced later in other code change.
go.work is said to be a file you rarely need to commit to the repository. You must be aware of what the impact of having go.work in parent paths would be, though.
--
References:
https://go.dev/doc/modules/managing-source: Discussion around repository setup
https://go.dev/ref/mod: Go Modules Reference
Side Note:
I have given a talk about this at Go Conference, hosted in Japan - you can find some demo code, slides, etc. here if you are curious to know more with examples.
In 2022, the best practice approach to having multiple modules within the same repository and importing them in other modules.
This is supported with a new "go module workspace".
Released with Go 1.18 and the new go work command.
See "Proposal: Multi-Module Workspaces in cmd/go" and issue 45713:
The presence of a go.work file in the working directory or a containing directory will put the go command into workspace mode.
The go.work file specifies a set of local modules that comprise a workspace.
When invoked in workspace mode, the go command will always select these modules and a consistent set of dependencies.
go.work file:
go 1.18
directory (
./baz // foo.org/bar/baz
./tools // golang.org/x/tools
)
replace golang.org/x/net => example.com/fork/net v1.4.5
You now have CL 355689
cmd/go: add GOWORK to go env command
GOWORK will be set to the go.work file's path, if in workspace mode
or will be empty otherwise.
I'm currently experimenting with golang's plugin system. A problem which I experienced in my testings is that the file size of the plugins is relativly big.
The application loading the plugin will be referenced as "host".
The host application itself is ~50MiB big since it is a web application and should be extended with plugin functionality.
I've implemented a small plugin loader to start the plugins up.
The plugins may use the already existing APIs in the application for example to access the database.
I've prepared a example plugin for this question. The plugin .so file size is ~39MiB. This gives me the reasonable suspicion that the plugin also contains source code from the host application.
Command used to create main.so:
go build -ldflags="-s -w" --buildmode=plugin main.go
Is it possible to "remove" the duplicated source code from the application to reduce file size since it is already loaded on runtime when the plugin gets loaded?
Plugin loader: https://github.com/jonasfranz/gitea/blob/feature/plugin/modules/plugins/loader.go
Example plugin: https://git.jonasfranz.software/JonasFranzDEV/giteaplugin
Source code is not included in plugins. But what is included in them is their dependencies, recursively. This is so because there is no guarantee that the main app that loads the plugin also contains the dependencies, so to ensure the viability of the plugin, its dependencies must be self-contained.
This does not cause problems if the main app also include the same dependencies (with the same version), they will only be "instantiated" once in the go runtime, for details, see How do Go plugin dependencies work?
What to do in order to reduce plugins' sizes? Besides removing the debug information (what you did), you should minimize the dependencies.
This may require redesign and major changes both in the plugin or in the app you wish to create the plugin for. For example, plugins should not refer to "implementation" packages, plugins should only refer to "interface" packages. If interfaces and implementations are not separated, this may not be possible (hence may it be required to change the main app too).
You may also try utilities that try to compress binaries, for details see: Shrink your Go binaries with this one weird trick
I have started learning go (1.7.4) and have a project which currently produces two executables. I have a directory structure as below following the standard go layout:
GOPATH=`pwd`
bin
src/
src/<project1>
src/<project1>/vendor
src/<project1>/glide.yaml
src/<project2>
src/<project2>/vendor
src/<project2>/glide.yaml
pkg/
Project 1 and project 2 share a lot of dependencies.
Is there a way to share the vendor directory between project1 and project2 and still pin the versions to ensure reproducible builds?
I don't want to duplicate the glide.yaml and vendor directories for each project as it bloats the build and violates DRY.
The pkg directory is the obvious the way to do this but unlike vendor I don't have a dependency manager tool like glide to ensure a specific version is used (see also my related question).
A possibly related issue is how this project is organised. I believe in go it would be more conventional for each project sub-directory to map to a single github repository. However, for my project I want to build at least two executables. I realise you can do this by using different package names but it confuses go and glide. I wrestled with getting this to work under a single project and decided/discovered it was easier to use the standard go layout and work two levels up. For example, an advantage is that "go build" etc. in the subdirectories just works without having to name the package. I can also have my build, test and package machinery at the top level operate on all projects and keep my go environment separate from any others.
The programs are not complex enough to warrant separate git repositories (even as submodules). If there is a pattern that makes this work it might render my original question moot.
It should be possible to have a shared vendor directory. The way I am doing it involves Go 1.11 and the new Go feature called modules. But I am pretty sure it should work with vendor and tools like glide and dep. To use dep/glide your directory structure might looks like this
- src
- projects
- project1
- project2
- vendor
- Glide.yaml
And you can build it either from the projects folder using go build -o p1 project1/*.go or from individual project folder using go build
The same structure, but outside of GOPATH will work for Go 1.11 modules. You would have to set the GO111MODULE variable to "on" or "auto". Mind you that go modules store dependencies in some other location and download them automatically during the build process when needed.
Note: glide github page recommends switching to dep as the more official tool
Edit: Just tested it with dep. It works for me.
I recommend look at new vendoring system - https://github.com/golang/go/wiki/Modules
It allows you to fix versions of packages used:
module github.com/my/thing
require (
github.com/some/dependency v1.2.3
github.com/another/dependency/v4 v4.0.0
)
In a project I'm working on, we auto generate the interfaces API in a folder called api/ which contains several sub-folders, where each of them has a pom file able to compile the content of the module.
project-root
- api
- module-api-1
- pom.xml
- module-api-2
- pom.xml
- module-api-3
- pom.xml
- module-api-4
- pom.xml
- build
- pom.xml
Basically the pom.xml triggers the code generator which then generates all the api/* modules. By the time I run maven clean install within the folder build/, the api folder is empty, because it will be filled by the code generator in the generate-code Maven phase.
Is there a way to tell the build/pom.xml to handle the modules inside api (the names are known) within the same build?
If I specify a <module> which does not exist, maven verify will complain.
Thanks
I believe the resolution depends on flexibility of the API's list
if the list of API-modules is dynamic, it's simply not possible to declare a dependency on a particular module - nobody knows them in advance. I would consider generation of source folders and adding them to a single module. As the result, you'll have single all-in-one module, which will contain all generated and compiled code. Other projects can use it as dependency
if the list of API-modules is fixed, their POM files, which declare GAVs, should not be generated. Then other projects can use any of them as dependencies, although their code is generated only during the build
If it was my Project, I would declare the references to the modules static in the pom (modules/ module-api-1 module-api-2 ...) and also have the module-Projects in a generated state so it theoretically could compile without generating the apis. So what I'm saying is - just treat these modules as full fledged module projects.
Then and I assume this is important for you, if you have a Change in the code that causes a Change in one or more apis, i would run the Generator. If you need to reflect this changed api in a repo, you can still just install the changed module.
I know this propably isnt what you wanted to do, but I'm pretty sure you'll have less Problems taking "the conservative way".
I am having trouble bringing in a subrepository in my project. The idea is that I bring a module in /arbitrary/folder and I want to include it on my current project located at /important/project
When using the GUI Import Module... from the menu the actual folders are copied into my project subfolder. Given that both projects are in different repositories there is a maintenance problem as for every update the files need to be manually reimported and then committed to the main project. Mercurial also doesn't allow you to have subrepos at any depth bigger than root.
What is the correct approach to solve this problem?
You're right that the GUI for Import Module makes a copy instead of importing it in-place. Until that functionality is improved, you'll need to set it up by hand.
In your module being imported, make sure it has a build.gradle file that's properly set up to build the module. Then in your application's settings.gradle file, include it like this:
include ':some_module'
project(':some_module').projectDir = new File('/path/to/module')
Then you can depend on it from another module in the usual way, either by adding a dependency through the Project Structure UI or by adding this to the module:
dependencies {
compile project(':some_module')
}