Suppose I have a monorepo and there are several individual golang services:
root
└── services
├── svc1
│ ├── go.mod
│ ├── go.sum
│ └── main.go
└── svc2
├── go.mod
├── go.sum
└── main.go
Whereas svc2 will depend on svc1 in the future. Also, it is a requirement that svc1 and svc2 can be released individually.
Is there a way I can publish the module indivdually? The go mod documentation only assume that there is only repo so it doesn't provide such flexibility.
To tag a release version for a module within a subdirectory of a repository, add the subdirectory as a prefix of the tag, like svc1/v0.1.0.
Per https://golang.org/ref/mod#vcs-version:
If a module is defined in a subdirectory within the repository …, then each tag name must be prefixed with the module subdirectory, followed by a slash. For example, the module golang.org/x/tools/gopls is defined in the gopls subdirectory of the repository with root path golang.org/x/tools. The version v0.4.0 of that module must have the tag named gopls/v0.4.0 in that repository.
Related
I am trying to learn Go modules dependency manager but getting an error while importing auto-generated proto definitions. Have been struggling with this for 2 days. Any help will be greatly appreciated.
Directory structure:
➜ ~/go-service-skeleton> tree .
.
├── protobuf
│ └── test_service
│ ├── test_service_config.pb.go
│ └── test_service_config.proto
├── src
│ └── test_service
│ ├── go.mod
│ ├── main.go
│ └── server
│ └── server.go
Error (last error is particularly concerning):
➜ ~/go-service-skeleton/src/test_service go build
bootstrap.go:11:2: no required module provides package github.com/Sirupsen/logrus; to add it:
go get github.com/Sirupsen/logrus
bootstrap.go:12:2: no required module provides package github.com/sirupsen/logrus; to add it:
go get github.com/sirupsen/logrus
bootstrap.go:13:2: no required module provides package gopkg.in/yaml.v2; to add it:
go get gopkg.in/yaml.v2
server/server.go:8:2: package protobuf/test_service is not in GOROOT
~/go-service-skeleton/src/test_service echo $GOPATH
/Users/****/go-service-skeleton:/Users/****/go-service-skeleton/protobuf
~/go-service-skeleton/src/test_service echo $GOROOT
Within your test_service module, the only dependencies visible to the go command are the ones declared in its go.mod file.
The simplest fix is to put all of the source code you need inside the main module. Note that per https://blog.golang.org/generate, “if the containing package is intended for import by go get, once the file is generated (and tested!) it must be checked into the source code repository to be available to clients.”
I have a project that looks similar to this tree:
. proj
├── README.md
├── go.mod
├── go.sum
├── subpackage
│ ├── one.go
│ ├── two.go
│ ├── three.go
│ └── four.go
├── main.go
In main.go I am importing a struct and some methods via github.com/username/proj/subpackage.
I am not using subpackage in any other projects (only this project).
Should I have a separate go.mod file within subpackage/ in addition to the go.mod file in project root? And consequently should the subpackage be listed in project-root's go.sum file?
If not, would I need it in the case where the subpackage directory/package is being used in a completely separate project?
A separate go.mod for subpackages is unnecessary. With such a go.mod file it is possible to separately tag the subpackage with a different version than the root package. This usecase is only needed in very specialized instances, where you want to have different versions for packages inside the same repository, so do not do that.
You do not have to modify go.sum manually at all. It should only be modified by the go-tooling (go get, go build), automatically.
Even if the subpackage is used inside another project by importing it, you do not have to give the subpackage a different version. When this subpackage is imported the go-tool will search for the root of the git repository, download the repository automatically with go get and make it available for importing.
Envs
GO111MODULE=on
golang:1.15.2-alpine
What I want to do
I implemented some features in a local project and I want to confirm the behavior before pushing
changes to the remote repo.
But my project can't access the new package.
Error text
When docker-compose run 'go build' I got his error.
Both 'github.com/Asuha-a/URLShortener/api/pb/url' and 'github.com/Asuha-a/URLShortener/api/utility' don't exist in the remote repo because it's not pushed yet.
main.go:10:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a), but does not contain package github.com/Asuha-a/URLShortener/api/pb/url
main.go:12:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a), but does not contain package github.com/Asuha-a/URLShortener/api/utility
Project architecture
I deleted some of the files look like not related to this issue.
$ tree
.
├── api
│ ├── Dockerfile
│ ├── go.mod
│ ├── go.sum
│ ├── main.go
│ ├── pb
│ │ ├── url // new package
│ │ ├── url_grpc.pb.go
│ │ ├── url.pb.go
│ │ └── url.proto
│ ├── services
│ │ ├── url
│ │ │ ├── db
│ │ │ │ ├── settings.go
│ │ │ │ └── url.go
│ │ │ ├── Dockerfile
│ │ │ ├── go.mod
│ │ │ ├── go.sum
│ │ │ └── main.go // failed building
│ └── utility // new package
│ ├── jwt.go
│ └── url.go
├── docker-compose.yml
What I tried
I found this issue.
Importing local changes of a package without pushing code in Go
That says I can access local packages by adding 'replace MODULE_URL => /PATH/TO/MODULE/DIR'.
I edited the 'api/services/url/go.mod' like this.
module github.com/Asuha-a/URLShortener/api/services/url
go 1.15
replace github.com/Asuha-a/URLShortener/api => ../../.
require (
github.com/jackc/pgx/v4 v4.9.2 // indirect
github.com/satori/go.uuid v1.2.0
golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9 // indirect
golang.org/x/text v0.3.4 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/grpc v1.33.2
gorm.io/datatypes v1.0.0
gorm.io/driver/postgres v1.0.5
gorm.io/gorm v1.20.6
)
But I got this error:
main.go:10:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a, replaced by ../../.), but does not contain package github.com/Asuha-a/URLShortener/api/pb/url
main.go:12:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a, replaced by ../../.), but does not contain package github.com/Asuha-a/URLShortener/api/utility
Then I tried the full path '/home/asuha/go/src/github.com/Asuha-a/URLShortener/api'.
But I got this error:
main.go:10:2: github.com/Asuha-a/URLShortener/api#v0.0.0-00010101000000-000000000000: replacement directory /home/asuha/go/src/github.com/Asuha-a/URLShortener/api does not exist
main.go:12:2: github.com/Asuha-a/URLShortener/api#v0.0.0-00010101000000-000000000000: replacement directory /home/asuha/go/src/github.com/Asuha-a/URLShortener/api does not exist
I want to know
How to fix this error?
Some basics first:
A module is a set of packages versioned together.
A module has a name declared in the go.mod, e.g. github.com/anyone/someproject
A package is imported by its import path (the import path basically is package identity).
A package belonging to a module must have an import path beginning with the module name. In the example above any package belonging to the module github.com/anyone/someproject must have an import path like e.g. github.com/anyone/someproject/whatever/hierarchy/pkgname
You never need to replace packages from the same module. replace is for replacing other modules.
You declared the module
module github.com/Asuha-a/URLShortener/api/services/url
That makes the name of your module "github.com/Asuha-a/URLShortener/api/services/url" and only packages with import path like module "github.com/Asuha-a/URLShortener/api/services/url/I/really/like/deeply/nested/stuff" would belong to that package.
Clearly you utility package (btw. don't do such packages, this is fundamental code smell!) doesn't belong to your module: You probably try to import it as "github.com/Asuha-a/URLShortener/api/utiliyt" but this is not part of the module "github.com/Asuha-a/URLShortener/api/services/url".
The fact that the folder utility is "below" the go.mod file in the filesystem doesn't overrule the fact that your module declares on its first line: "All my packages have import paths starting with github.com/Asuha-a/URLShortener/api/services/url! Anything else does not belong to me!"
You probably should name your module at least module github.com/Asuha-a/URLShortener/api (if not module github.com/Asuha-a/URLShortener).
I urge you to (re-)read "How to Write Go Code" and stick to it.
Takeaway:
Modules are really just a set of packages with the same import path prefix (the module name) versioned together.
I have the following project structure
.
├── README.md
├── protos
│ ├── my-proto-output.pb.go
│ └── my-proto.proto
└── grpc-backend
├── client
│ └── client.go
├── go.mod
├── go.sum
├── main.go
└── a-submodule
Inside grpc-backend/main.go I have:
package main
import (
pb "github.com/my-user/my-repo/protos"
"google.golang.org/api/option"
"google.golang.org/grpc"
)
Inside go.mod I have
go 1.13
require (
google.golang.org/api v0.14.0
google.golang.org/grpc v1.21.1
)
module github.com/my-user/my-repo/grpc-backend
However, when I run cd grpc-backend && go build I get
go: finding github.com/my-user/my-repo latest
go: finding github.com/my-user/my-repo/protos latest
go: downloading github.com/my-user/my-repo v0.0.0-20200103231607-5a754c449f99
verifying github.com/my-user/my-repo#v0.0.0-20200103231607-5a754c449f99: github.com/my-user/my-repo#v0.0.0-20200103231607-5a754c449f99: reading https://sum.golang.org/lookup/github.com/my-user/my-repo#v0.0.0-20200103231607-5a754c449f99: 410 Gone
I don't have any tags like that, I'm in a private repo only commiting to master at the moment. Furthermore, proto's are in this same git repo under the package myrepo.
Questions:
Did I organize this incorrectly?
Where did that version come from? It's not in go.sum
Your module does not include the protos package, and thus, the generated grpc source. Move your go.mod one level up, and change the module name to github.com/my-user/my-repo so the module includes all packages.
I have some problems with how Go importing and modules works.
I have a project with this structure:
My-Project
|- Program 1
|- main.go
|- go.mod
|- Program 2
|- main.go
|- go.mod
|- Support
|- go_file.go
The two programs have different module "definition". I need to include the "Support" folder, which is shared between the two programs. I tried
import "My-Project/Support"
inside Program1's main.go, but this is not working, because Support can’t be resolved. How can I do it?
As given, your Support/go_file.go is not in any module so it's not importable using Go modules.
Either use a single module for all three packages or use three separate modules (i.e. put Support in it's own module). You could also use sub-modules.
Single module:
For a single module you'd do something like:
└── My-Project
├── go.mod
├── Program\ 1
│ └── main.go
├── Program\ 2
│ └── main.go
└── Support
└── go_file.go
And the project level go.mod would be something like:
module github.com/My-Name/My-Project
(substitute with appropriate module name) and then both main packages would import the support package with:
import "github.com/My-Name/My-Project/Support"
Note Go packages normally do not start with capitals like this.
More idiomatic for such a setup with a single repository which mixes packages and commands is a layout that puts commands into a cmd subdirectory (e.g. see the golang.org/x/tools layout). This would look something like this:
└── My-Project
├── cmd
│ ├── program1
│ │ └── main.go
│ └── program2
│ └── main.go
├── go.mod
├── internal
│ └── support
│ └── support.go
└── other-package
└── some-other-package.go
Here the support package is in an internal sub-directory which makes it non-importable from outside My-Project where-as the other-package is importable by anyone.
Again, go.mod would define the module name which sets the prefix to be used for all import statements.
Separate modules:
To use separate modules you'd do something like:
└── My-Project
├── Program\ 1
│ ├── go.mod
│ └── main.go
├── Program\ 2
│ ├── go.mod
│ └── main.go
└── Support
├── go_file.go
└── go.mod
Here My-Project/Program 1/go.mod would be something like:
module "github.com/My-Name/My-Project/Program 1"
require (
github.com/My-Name/My-Project/Support
)
The imports would be the same as for the single module example.
Similarly for Program 2. For Support/go.mod it'd look something like:
module github.com/My-Name/My-Project/Support
For local development you'll probably also want/need a replace directive in each program's go.mod to use your local working copy of the support module (go mod edit -replace github.com/My-Name/My-Project/Support=../Support should add this).