I am developing multiple microservices, which require different modules (which shall be available like modules on github, but private)
My first tests with Go were all located within one package, which gets quite messy after some time
I'm coming from the Java side of programming - with loads of packages - which keep stuff clear and clean.
(This also works with public modules like on github.com/xyz/module1 github.com/xyz/module2 github.com/xyz/module3)
I just need this for private modules - how can I do so?
This is what I tried yet:
Directory of my go sources:
C:\my\path\to\go\src\
In this dir I have multiple subdirs containing modules (actually more than listed here)
my-module1
my-module2
my-module3
For each folder I called go mod init but I get the message that
package my-module1/util is not in GOROOT (c:\go\src\my-module1\util)
which is obviously right, as my private libraries reside in C:\my\path\to\go\src\
Importing packages from github with go get ... is working without troubles (those packages will be loaded but copied to c:\go\src)
Working with all files in one folder works but is not the desired solution (I need to create multiple microservices therefore I want to be able to create different projects with custom executeables and or tests)
What am I doing wrong?
If more information is needed I will provide it - just let me know what
NOTE: packages without go file in package main cannot be installed via go install. This system looks pretty confusing to me - as modules cannot be found...
finally i created a multimodule
means i have a root folder (lets call it root) which contains a go.mod file
this defines the root module
it requires all my submodules
and it replaces the absolute path into a relative.
root
* aa
* ab
* bc
main gets defined as
module root
go 1.15
require (
root/aa v0.0.0
root/ab v0.0.0
root/bc v0.0.0
)
replace (
root/aa => ./aa
root/ab => ./ab
root/bc => ./bc
)
all the submodules have a very simple definition (this is equal for all, as long as you do not have any dependencies)
module root/aa
Related
I ran into a repo that seems to be a Go module, but there's no go.mod file in it: github.com/confluentinc/confluent-kafka-go.
Is it ok for a go module to have no go.mod file with dependencies, or the authors of that library just didn't migrate to modules yet?
Dependency modules do not need to have explicit go.mod files.
The “main module” in module mode — that is, the module containing the working directory for the go command — must have a go.mod file, so that the go command can figure out the import paths for the packages within that module (based on its module path), and so that it has a place to record its dependencies once resolved.
In addition, any modules slotted in using replace directives must have go.mod files (in order to reduce confusion due to typos or other errors in replacement paths).
However, in general a module that lacks an explicit go.mod file is valid and fine to use. Its effective module path is the path by which it was required, which can be a bit confusing if the same repository ends up in use via multiple paths. Since a module with no go.mod file necessarily doesn't specify its own dependencies, consumers of that module will have to fill in those dependencies themselves (go mod tidy will mark them as // indirect in the consumer's go.mod file).
SHORT SUMMARY OF THE DISCUSSION:
The answer is "No"!
This project contains a set of go packages, but it is not a Go module as it doesn't contain go.mod file (although, it used to be a multi-module repo (Go) previously).
go get can run in both ways: module-aware mode and legacy GOPATH mode (as of Go 1.16).
To read more about this, refer to docs by using the go command:
$ go help gopath-get
and
$ go help module-get
It'd tell about how go get works in both cases.
Also, I noticed that it can download any repository and would treat it as a Go package, even if it contains an arbitrary Python project.
I did a simple test to demonstrate the same:
$ go get github.com/mongoengine/mongoengine
And it surprisingly worked.
Modules are defined by their go.mod file. Without a go.mod file, it is not a module.
See this from the Go Modules Reference
A module is a collection of packages that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or from module proxy servers.
A module is identified by a module path, which is declared in a go.mod file, together with information about the module's dependencies. The module root directory is the directory that contains the go.mod file.
And
A module is defined by a UTF-8 encoded text file named go.mod in its root directory.
What I essentially have is a mono-repo, which doesn't have a go.mod at the root level. There are multiple directories inside this mono-repo, with each of them having their own go.mod files. I'll refer to them as sub-modules.
Now, I've figured out a way to be able to access the sub-modules independently (versioned) in a completely different codebase. The issue I'm facing now is, to dis-allow the import of the entire mono-repo, using:
go get link.to/mono-repo#commit_id ------> A
and only allow import using :
go get link.to/mono-repo/sub_mod1#v0.x.y
go get link.to/mono-repo/sub_mod2#v0.y.z
The command A is able to fetch the entire repo, and then can be used to access the inner modules. Is there any way to stop this from being allowed?
I tried a few things, like:
Added non-compilable code in a file noCompile.go at the root level. On go get..., the compilation error is printed, but use of the inner modules still works fine.
Added an init() function in the same noCompile.go file, which just calls panic(). This init function is not being executed, as the root directory is never directly accessed, only the inner modules are.
Is there ANY way to achieve what I'm intending to?
Any directory that contains its own go.mod file is excluded from the module in the parent directory. So if you go get link.to/mono-repo#commit_id, that should not contain the packages link.to/mono-repo/sub_mod1 or link.to/mono-repo/sub_mod2 (assuming that they exist and have their own go.mod files).
I suspect that you were observing imports like import "link.to/mono-repo/sub_mod2/some/package/here" to resolve not because of that initial go get, but rather because the go command was resolving the missing imports (and adding the missing dependencies) automatically; see https://golang.org/ref/mod#go-mod-file-updates.
As of Go 1.16 (released today!), most go commands no longer implicitly modify the go.mod file, so the fact that the module at the repo root does not include the contents of nested modules should hopefully be clearer.
I can not find a way to factor out some code from main.go into a local package when using Go modules (go version >= 1.11) outside of $GOPATH.
I am not importing any external dependencies that need to be included into go.mod, I am just trying to organize locally the source code of this Go module.
The file main.go:
package main
// this import does not work
import "./stuff"
func main() {
stuff.PrintBaz()
}
The file ./stuff/bar.go (pretending to be a local package):
package stuff
import "log"
type Bar struct {
Baz int
}
func PrintBaz() {
baz := Bar{42}
log.Printf("Bar struct: %v", baz)
}
The file go.mod (command go mod init foo):
module foo
go 1.12
When executing go run main.go:
If I import "./stuff", then I see build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
If I import "stuff", then I see build command-line-arguments: cannot load stuff: cannot find module providing package stuff.
If I import stuff "./stuff" with a package alias, then I see again: build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
I can not find a way to make local packages work with go modules.
What's wrong with the code above?
How can I import local packages into other Go code inside a project defined with Go modules (file go.mod)?
Module structure
The most common and easiest approach is:
Use a single go.mod per repository, and
Place the single go.mod file in the repository root, and
Use the repository name as the module path declared in the module line in the go.mod
(If you are using a custom import path such as me.io/mymod rather than using a VCS host based import path, then you would use the custom import path instead of the repository name in your go.mod).
For example, if your repo is github.com/my/repo, then you would place a single go.mod in the repo root, with the first line reading module github.com/my/repo. That can be created by cd'ing to the repo root and running go mod init github.com/my/repo. When working with the repo locally, the repo directory can be located wherever is convenient on your filesystem. (If you do not have a repo, see below).
Following this helps you stay on the happy path with modules, and it avoids multiple subtleties.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
There is much more about multi-module repositories in the "Multi-module Repositories" FAQ section on the modules wiki. Those 6 or so FAQs in that section should be read in their entirety by anyone considering veering off the recommendation above.
Arranging packages within a module
Once you have set up your go.mod, you can arrange your packages in directories however you see fit in directories underneath the directory containing the go.mod, as well as in the directory with the go.mod. Three good articles about how to arrange your code in packages:
https://rakyll.org/style-packages/
https://medium.com/#benbjohnson/standard-package-layout-7cdbc8391fc1#.ds38va3pp
https://www.goinggo.net/2017/02/design-philosophy-on-packaging.html
Those articles are classics that pre-date the introduction of modules, but the philosophies in them still apply to how to arrange your packages within a module.
Importing other packages in the same module
When importing another package with modules, you always use the full path including the module path. This is true even when importing another package in the same module. For example, if a module declared its identity in its go.mod as module github.com/my/repo, and you had this organization:
repo/
├── go.mod <<<<< Note go.mod is located in repo root
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
Then pkg1 would import its peer package as import "github.com/my/repo/pkg2". Note that you cannot use relative import paths like import "../pkg2" or import "./subpkg". (This is part of what OP hit above with import "./stuff").
Modules vs. repositories vs. packages vs. import paths
A Go module is a collection of related Go packages that are versioned together as a single unit. Modules record precise dependency requirements and create reproducible builds.
Summarizing the relationship between repositories, modules, and packages:
A repository contains one or more Go modules (most often exactly one module in the repository root).
Each module contains one or more Go packages.
Each package consists of one or more Go source files that all reside in a single directory.
Each Go source code file:
declares its own package with a package foo statement, which must be consistent for all Go files in a single package directory.
automatically has access to other Go source code in the same package, without explicitly importing its own package.
imports code from another package via an import path supplied in an import statement such as import "github.com/my/repo/pkg1". The import path always starts with the module path of that package, regardless of whether that package is in the same module or a different module.
A single module can be used to build multiple executable binaries, but the func main() {...} entry point for each binary must be in its own package main, with each package main in a distinct directory.
Local-only modules
You can create, build and run modules without ever publishing them or placing them on a VCS server.
A module is defined by a tree of Go source files with a go.mod file in the tree's root directory, so you can pick a directory on your local filesystem to serve as the root of your module and place your go.mod there. Everything described above still applies, except there does not need to be a repo.
For example:
some-directory/
├── go.mod <<<< go.mod at module root declaring 'module example.com/foo'
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
If the module path in the first line of that go.mod reads module example.com/foo, then pkg1 would import its peer package as import "example.com/foo/pkg2".
Of course, you can start with a local-only module without a repo, then later start tracking it in a local repo or publish it to a VCS server.
Even for local-only modules, it is a good idea to give the module a globally unique name, such as go mod init my.org/foo, or go mod init github.com/some/repo as if it will be published someday, even if that repo does not yet exist. (If you are in a hurry, it is possible to use a shorter module name without a dot or leading hostname component, such as go mod init temp or go mod init m, but it won’t be fetchable by the go command, and you run the risk of colliding with a current or future standard library package or otherwise seeing confusing error messages later).
If you have multiple local modules, you can point them at each other via replace directives or via the Go 1.18+ workspace feature, though those are more complex workflows than placing everything for a given project in a single module.
Next steps
If you are new to Go modules, it is worthwhile to go through the official "Create a Go Module" tutorial from the Go project. This will likely save you time overall.
If you are stuck right now with an error message related to modules or import paths, there's a very good chance the error will make much more sense after completing the tutorial.
First you have to choose a name for your project and write it to go.mod file. This name belongs to root directory of the project. Each new package you create must be located inside its own subdirectory and its name should match directory name.
go.mod:
module myprojectname
or (preferred way, see #thepudd's answer for details)
module github.com/myname/myproject
Then import your project's packages like:
import myprojectname/stuff
or
import github.com/myname/myproject/stuff
Files of package stuff should be located inside project's stuff directory. You name these files as you like.
Also it's possible to create deeper project structure. For instance, you decided to separate source code files from other ones (like app configs, docker files, static files, etc...). Let's move stuff directory inside pkg, every go file inside pkg/stuff still have stuff package name. To import stuff package just write:
import myprojectname/pkg/stuff
Nothing stops you from creating more levels in the hierarchy like github.com/myuser/myproject/pkg/db/provider/postgresql, where:
github.com/myuser/myproject - project name.
postgresql - package name.
pkg/db/provider/postgresql - path to the package relative to project's root.
You can read more about go modules here: https://github.com/golang/go/wiki/Modules
Check out this repository to get useful information about various patterns used in project organizing: https://github.com/golang-standards/project-layout If you go inside pkg directory you will find out which open source projects use pkg directory in their structure.
I'm trying to understand how to organize my golang project using go1.11 modules. I tried several options, but none of them worked.
I have some code in the main package under the application folder and a local package that the main package uses.
$GOPATH
+ src
+ application/
+ main/
+ main.go
+ otherFileUnderMainPackage.go
+ aLocalPackage/
+ someCode.go
+ someCode_test.go
+ someMoreCode.go
+ someMoreCode_test.go
Files in the main package, imports ../aLocalPackage. When I compile by go build main/*.go it's working.
Then, I ran go mod init application: V.0.9.9 and got the go.mod file, but the build always fails. I always get error about not finding the local package: build application:V0.9.9/main: cannot find module for path _/.../src/application/aLocalPackage. I also tried to place the local package right under src/, place it under main/ etc. but none of these methods worked for me.
What is the way to use modules and local packages?
Thanks.
Relative import paths are not supported in module mode. You will need to update your import statements to use a full (absolute) import path.
You should also choose a module name other than application. Your module path should generally begin with a URL prefix that you control — either your own domain name, or a unique path such as github.com/$USER/$REPO.
I had some problems working with local packages myself.
There are two tricks to make it work:
you run "go build" in the package directory
This compiles the package and places it in the build cache.
This link about code organisation in go explains more.
You can identify where the cache is using:
>go env GOCACHE
/home/<user>/.cache/go-build
Import using a path relative to the project
I puzzled loads over what the correct import path was and finally discovered that go doc or go list will tell you.
>go doc
package docs // import "tools/src/hello/docs"
>go list
tools/src/hello/docs
For example. I have a hello world API project and was using swaggo to generate documentation which it does in a docs sub-directory.
To use it I add an import:
_ "tools/src/hello/docs"
For my case the _ is important as docs is not used directly but we its init() function to be invoked.
Now in hello/main.go I can add "tools/src/hello/docs" and it will import the correct package.
The path is relative to the location of go.mod if you have one.
I have tools/ here as I have a go.mod declaring "modules tools".
Modules are a different kettle of fish - see https://github.com/golang/go/wiki/Modules.
Recent versions of go (1.11 and later) can create a go.mod file which you may use to fix the version of a module that is used and avoid go's crazy default behaviour of just downloading the latest version of any package you import.
I have written a blogpost on how to start your first Go project using modules.
https://marcofranssen.nl/start-on-your-first-golang-project/
In general it boils down to just create a new folder somewhere on your system (doesn't have to be in GOPATH).
mkdir my-project
cd my-project
go mod init github.com/you-user/my-project
This will create the go.mod file. Now you can simply create your project layout and start building whatever you like.
Maybe one of my other blogs can inspire you a bit more on how to do things.
https://marcofranssen.nl/categories/software-development/golang/
How to setup project structure to produce library and cli with same name in
single repository?
Suppose my project name is project. I want to make it importable with name
project and have executable binary with name project when installed with
go get. My setup currently is like this:
host.com/project/
project/
main.go
core/
project.go
Then, when installed with:
go get host.com/project/project
it installs project as executable which pulls core as dependency. In
core/project.go file, the package has this:
package project
The problem is it is imported with:
import (
"host.com/project/core"
)
And it exports project as name space not core which violates go's convention.
How can I do this?
This isn't a convention, and you may do however you like, but it's a common practice to have a cmd folder in the project root, and all the executables go under that, each in is own folder.
So a good solution in your case could be simply:
host.com/project/
cmd/
project/
project.go
filea.go
fileb.go
In filea.go and fileb.go the package declaration should be:
package project
In cmd/project/project.go package and import declaration should be:
package main
import "host.com/project"
Of course if your project is more complex, you may create further packages under the project root, and it may have multiple commands (multiple main packages), e.g.:
host.com/project/
cmd/
project/
project.go
prjtool/
prjtool.go
packagex/
x.go
packagey/
y.go
filea.go
fileb.go
An example following this layout is the very popular Go Delve debugger (4.5k stars currently), available at https://github.com/derekparker/delve.
A very complex repository example is the golang.org/x/tools package which is a collection of multiple, essential Go tools, available at https://github.com/golang/tools/. Its cmd folder contains more than 20 subfolders (commands), many of the tools (main packages) even consist of multiple .go files (but each using package main declaration, forming a single package of the executable).
Also check out the following related question: What is a sensible way to layout a Go project