Problem with modules and importing folders - go

I have some problems with how Go importing and modules works.
I have a project with this structure:
My-Project
|- Program 1
|- main.go
|- go.mod
|- Program 2
|- main.go
|- go.mod
|- Support
|- go_file.go
The two programs have different module "definition". I need to include the "Support" folder, which is shared between the two programs. I tried
import "My-Project/Support"
inside Program1's main.go, but this is not working, because Support can’t be resolved. How can I do it?

As given, your Support/go_file.go is not in any module so it's not importable using Go modules.
Either use a single module for all three packages or use three separate modules (i.e. put Support in it's own module). You could also use sub-modules.
Single module:
For a single module you'd do something like:
└── My-Project
├── go.mod
├── Program\ 1
│   └── main.go
├── Program\ 2
│   └── main.go
└── Support
└── go_file.go
And the project level go.mod would be something like:
module github.com/My-Name/My-Project
(substitute with appropriate module name) and then both main packages would import the support package with:
import "github.com/My-Name/My-Project/Support"
Note Go packages normally do not start with capitals like this.
More idiomatic for such a setup with a single repository which mixes packages and commands is a layout that puts commands into a cmd subdirectory (e.g. see the golang.org/x/tools layout). This would look something like this:
└── My-Project
├── cmd
│   ├── program1
│   │   └── main.go
│   └── program2
│   └── main.go
├── go.mod
├── internal
│   └── support
│   └── support.go
└── other-package
└── some-other-package.go
Here the support package is in an internal sub-directory which makes it non-importable from outside My-Project where-as the other-package is importable by anyone.
Again, go.mod would define the module name which sets the prefix to be used for all import statements.
Separate modules:
To use separate modules you'd do something like:
└── My-Project
├── Program\ 1
│   ├── go.mod
│   └── main.go
├── Program\ 2
│   ├── go.mod
│   └── main.go
└── Support
├── go_file.go
└── go.mod
Here My-Project/Program 1/go.mod would be something like:
module "github.com/My-Name/My-Project/Program 1"
require (
github.com/My-Name/My-Project/Support
)
The imports would be the same as for the single module example.
Similarly for Program 2. For Support/go.mod it'd look something like:
module github.com/My-Name/My-Project/Support
For local development you'll probably also want/need a replace directive in each program's go.mod to use your local working copy of the support module (go mod edit -replace github.com/My-Name/My-Project/Support=../Support should add this).

Related

Golang modules: Import proto definitions

I am trying to learn Go modules dependency manager but getting an error while importing auto-generated proto definitions. Have been struggling with this for 2 days. Any help will be greatly appreciated.
Directory structure:
➜ ~/go-service-skeleton> tree .
.
├── protobuf
│   └── test_service
│   ├── test_service_config.pb.go
│   └── test_service_config.proto
├── src
│   └── test_service
│   ├── go.mod
│   ├── main.go
│   └── server
│   └── server.go
Error (last error is particularly concerning):
➜ ~/go-service-skeleton/src/test_service go build
bootstrap.go:11:2: no required module provides package github.com/Sirupsen/logrus; to add it:
go get github.com/Sirupsen/logrus
bootstrap.go:12:2: no required module provides package github.com/sirupsen/logrus; to add it:
go get github.com/sirupsen/logrus
bootstrap.go:13:2: no required module provides package gopkg.in/yaml.v2; to add it:
go get gopkg.in/yaml.v2
server/server.go:8:2: package protobuf/test_service is not in GOROOT
~/go-service-skeleton/src/test_service echo $GOPATH
/Users/****/go-service-skeleton:/Users/****/go-service-skeleton/protobuf
~/go-service-skeleton/src/test_service echo $GOROOT
Within your test_service module, the only dependencies visible to the go command are the ones declared in its go.mod file.
The simplest fix is to put all of the source code you need inside the main module. Note that per https://blog.golang.org/generate, “if the containing package is intended for import by go get, once the file is generated (and tested!) it must be checked into the source code repository to be available to clients.”

My app failed to import another local package without pushing changes to remote repo

Envs
GO111MODULE=on
golang:1.15.2-alpine
What I want to do
I implemented some features in a local project and I want to confirm the behavior before pushing
changes to the remote repo.
But my project can't access the new package.
Error text
When docker-compose run 'go build' I got his error.
Both 'github.com/Asuha-a/URLShortener/api/pb/url' and 'github.com/Asuha-a/URLShortener/api/utility' don't exist in the remote repo because it's not pushed yet.
main.go:10:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a), but does not contain package github.com/Asuha-a/URLShortener/api/pb/url
main.go:12:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a), but does not contain package github.com/Asuha-a/URLShortener/api/utility
Project architecture
I deleted some of the files look like not related to this issue.
$ tree
.
├── api
│   ├── Dockerfile
│   ├── go.mod
│   ├── go.sum
│   ├── main.go
│   ├── pb
│   │   ├── url // new package
│   │      ├── url_grpc.pb.go
│   │      ├── url.pb.go
│   │      └── url.proto
│   ├── services
│   │   ├── url
│   │   │   ├── db
│   │   │   │   ├── settings.go
│   │   │   │   └── url.go
│   │   │   ├── Dockerfile
│   │   │   ├── go.mod
│   │   │   ├── go.sum
│   │   │   └── main.go // failed building
│   └── utility // new package
│   ├── jwt.go
│   └── url.go
├── docker-compose.yml
What I tried
I found this issue.
Importing local changes of a package without pushing code in Go
That says I can access local packages by adding 'replace MODULE_URL => /PATH/TO/MODULE/DIR'.
I edited the 'api/services/url/go.mod' like this.
module github.com/Asuha-a/URLShortener/api/services/url
go 1.15
replace github.com/Asuha-a/URLShortener/api => ../../.
require (
github.com/jackc/pgx/v4 v4.9.2 // indirect
github.com/satori/go.uuid v1.2.0
golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9 // indirect
golang.org/x/text v0.3.4 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/grpc v1.33.2
gorm.io/datatypes v1.0.0
gorm.io/driver/postgres v1.0.5
gorm.io/gorm v1.20.6
)
But I got this error:
main.go:10:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a, replaced by ../../.), but does not contain package github.com/Asuha-a/URLShortener/api/pb/url
main.go:12:2: module github.com/Asuha-a/URLShortener/api#latest found (v0.0.0-20201114002535-3dac2ebd322a, replaced by ../../.), but does not contain package github.com/Asuha-a/URLShortener/api/utility
Then I tried the full path '/home/asuha/go/src/github.com/Asuha-a/URLShortener/api'.
But I got this error:
main.go:10:2: github.com/Asuha-a/URLShortener/api#v0.0.0-00010101000000-000000000000: replacement directory /home/asuha/go/src/github.com/Asuha-a/URLShortener/api does not exist
main.go:12:2: github.com/Asuha-a/URLShortener/api#v0.0.0-00010101000000-000000000000: replacement directory /home/asuha/go/src/github.com/Asuha-a/URLShortener/api does not exist
I want to know
How to fix this error?
Some basics first:
A module is a set of packages versioned together.
A module has a name declared in the go.mod, e.g. github.com/anyone/someproject
A package is imported by its import path (the import path basically is package identity).
A package belonging to a module must have an import path beginning with the module name. In the example above any package belonging to the module github.com/anyone/someproject must have an import path like e.g. github.com/anyone/someproject/whatever/hierarchy/pkgname
You never need to replace packages from the same module. replace is for replacing other modules.
You declared the module
module github.com/Asuha-a/URLShortener/api/services/url
That makes the name of your module "github.com/Asuha-a/URLShortener/api/services/url" and only packages with import path like module "github.com/Asuha-a/URLShortener/api/services/url/I/really/like/deeply/nested/stuff" would belong to that package.
Clearly you utility package (btw. don't do such packages, this is fundamental code smell!) doesn't belong to your module: You probably try to import it as "github.com/Asuha-a/URLShortener/api/utiliyt" but this is not part of the module "github.com/Asuha-a/URLShortener/api/services/url".
The fact that the folder utility is "below" the go.mod file in the filesystem doesn't overrule the fact that your module declares on its first line: "All my packages have import paths starting with github.com/Asuha-a/URLShortener/api/services/url! Anything else does not belong to me!"
You probably should name your module at least module github.com/Asuha-a/URLShortener/api (if not module github.com/Asuha-a/URLShortener).
I urge you to (re-)read "How to Write Go Code" and stick to it.
Takeaway:
Modules are really just a set of packages with the same import path prefix (the module name) versioned together.

Import package from another module

I have to create a desktop app on top of existing Go backend code, since this requirement and my knowledge in web development I'm using Wails.
With Wails CLI I generated the desktop app project in a subfolder of the back end main project.
This is how the folder structure looks like:
backend
├── [...]
├── logger
│   └── logger.go
├── gui
│   └── desktopApp
│   ├── build
│   │   └── desktopApp
│   ├── frontend
│   │   └── [...]
│   ├── go.mod
│   ├── go.sum
│   ├── main.go
│   └── project.json
├── main.go
├── go.mod
└── go.sum
The logger has been imported in the desktopApp and everything works during the development and testing via browser. But when I build the desktop app with the command wails build from the desktopApp folder I have got the following error:
go: <domain>/<owner>/backend/logger: unrecognized import path "<domain>/<owner>/backend"
I imported the backend in the desktop app like this:
/backend/gui/desktopApp/go.mod
module desktopApp
go 1.15
require (
<domain>/<owner>/backend v0.0.0
)
./gui/desktopApp/main.go
package main
import (
"github.com/leaanthony/mewn"
"github.com/wailsapp/wails"
"<domain>/<owner>/backend/logger"
)
// Using the logger package normally
How can I fix my issue?
I see that you use go-module and store code in a repo with sub-module
The problem is that you forgot add "desktopApp"-prefix
import (
"github.com/leaanthony/mewn"
"github.com/wailsapp/wails"
"desktopApp/backend/backendPackage"
)
Right way use modules👇🏼
Fix go.mod of desktopApp module
module <domain>/<owner>/desktopApp
go 1.15
fix backends import , because it belongs to desktopApp
import (
"github.com/leaanthony/mewn"
"github.com/wailsapp/wails"
"<domain>/<owner>/desktopApp/backend/backendPackage"
)
I solved my issue changing go.mod file of the nested desktop app like this:
module desktopApp
go 1.15
require (
<domain>/<owner>/backend/logger
)
// Added this line
replace <domain>/<owner>/backend/logger => ../../

Using go modules with a typical directory structure, can't seem to get it working

So I created a simple test project to demonstrate the issue I'm having in a much larger code base here https://github.com/cleblanc189/test.go
.
├── go.mod
├── main.exe
├── main.go
├── server
│   ├── v1beta1
│   │   ├── foo.go
│   │   └── go.mod
│   └── v1ga
│   ├── foo.go
│   └── go.mod
└── service
├── v1beta1
│   ├── foo.go
│   └── go.mod
└── v1ga
├── foo.go
└── go.mod
I'd like to be able to reference service/v1beta1 from server/v1beta1 but I'm getting this error
go: test.go/src/server/v1beta1#v0.0.0 requires
test.go/src/service/v1beta1#v0.0.0: unrecognized import path "test.go/src/service/v1beta1" (https fetch: Get https://test.go/src/ser
vice/v1beta1?go-get=1: dial tcp: lookup test.go: no such host)
In foo.go from server/v1beta1 I import like this
svc "test.go/src/service/v1beta1"
and have a go.mod file like below;
require test.go/src/service/v1beta1 v0.0.0
replace test.go/src/service/v1beta1 => ../../service/v1beta1
When I try to cd src && go mod tidy I get that error above. Also if anyone knows should I even have a go.mod above the src directory?
Appreciate any help anyone can offer. This is maddening.
Your go.mod has to declare the module path/name, typically like this:
module test.go
Then all the imports in your code will be taken relative to that. When you import something relative to your declared module name, the go tool will automatically pick up the code from the right place locally.
Not sure where your src comes into play though... is everything in folder src? See this post and accompanying code for a complete example you could start from.

Golang project directory structure

I have some confusion regarding Golang directory structure.
Base on the book The Way to Go by Ivo Balbaert, project code should be placed into src, and recommends the following directory structure.
├──src/
| ├──main.go
| ├──say/
| | ├──say.go
| | ├──say_test.go
├──bin/
| ├──say
└──pkg/
└──linux_amd64/
└──say.a
but I found that many packages in github.com, have no src directory.
For example:
https://github.com/facebookgo/grace
https://github.com/astaxie/beego
So, I don't know whether src directory is needed.
I have some project, their have inter-dependency.
They are managed in a private GitLab repository.
How can I organized them?
This article by Ben Johnson has guided me on this when I was starting with Go.
It's generally good to start with something like this (assuming you are inside your project directory like $GOPATH/src/myproject:
├──cmd/ -- this is where you compose several packages in to main package
| ├──foo -- an example would be `foo`
| | ├──main.go
├──pkg/ -- this is where put your reusable packages
| ├──pkg1 -- reusable package 1
| ├──pkg2 -- reusable package 2
├──otherpackage1
| ├── ...
├──otherpackage2
| ├── ...
You can have a look at this example from go-kit for this kind of project structure.
Sometimes it will depend on your needs. On our workflow, we are using a hot code reload tool called fresh, so we needed to put the main.go on the project root so that the tool can detect all the file changes and rebuild the source code.
├──app/
| ├──app.go
├──model/ --
| ├──model.go
├──store
| ├──store.go
├──main.go -- this is where the app starts
├──...
On the app.go package, I have something like func Run() error which starts the application. On the main.go, I am just calling the function:
...
func main(){
log.Fatal(app.Run())
}
Now there are new ways to organize Go projects.
The GitHub golang-standards repository says:
This is a basic layout for Go application projects. It represents the
most common directory structure with a number of small enhancements
along with several supporting directories common to any real world
application.
This project layout is intentionally generic and it doesn't try to
impose a specific Go package structure.
Or you can follow this slides:
$GOPATH/
src/
github.com/user/repo/
mypkg/
mysrc1.go
mysrc2.go
cmd/mycmd/
main.go
bin/
mycmd
Here is another project layout sample Simple Go project layout with modules
├── LICENSE
├── README.md
├── config.go
├── go.mod
├── go.sum
├── clientlib
│ ├── lib.go
│ └── lib_test.go
├── cmd
│ ├── modlib-client
│ │ └── main.go
│ └── modlib-server
│ └── main.go
├── internal
│ └── auth
│ ├── auth.go
│ └── auth_test.go
└── serverlib
└── lib.go
Some answer points out the go standard layout, however, there is one issue this is not a standard Go project layout by Russ Cox
There are two problems with this GitHub repo:
it claims to host Go standards and does not, in the sense that these are in no way official standards
the project-layout standard it puts forth is far too complex and not a standard
Regarding "why not tell us the standard Go project layout and we'll update the doc?", that only addresses point 2. If there really were standards, they would be in the main Go project doc tree. The standard for project layout would also be a lot shorter. I appreciate your trying to provide a useful resource, but calling it 'golang-standards' is claiming more than it is.
But for the record, the minimal standard layout for an importable Go repo is really:
Put a LICENSE file in your root
Put a go.mod file in your root
Put Go code in your repo, in the root or organized into a directory tree as you see fit
That's it. That's the "standard".
Update at 11/30/2021
Here are a summary from How to structure Go code
Before we start
doc.go file puts the general description of the package
Readme file a general overview of this project
When you have more documentation to provide put them into the docs folder
For linting use golangci-lint. Enable all linters that seem to be reasonable for your project
The flat structure (single package)
courses/
main.go
server.go
user_profile.go
lesson.go
course.go
When to create a new package?
When you have more than one way of starting your application
When you want to extract more detailed implementation
When you started to add a common prefix to closely related things
Modularisation
Organising by kind
.
├── handlers
│ ├── course.go
│ ├── lecture.go
│ ├── profile.go
│ └── user.go
├── main.go
├── models
│ ├── course.go
│ ├── lecture.go
│ └── user.go
├── repositories
│ ├── course.go
│ ├── lecture.go
│ └── user.go
├── services
│ ├── course.go
│ └── user.go
└── utils
└── stings.go
Organising by components
.
├── course
│ ├── httphandler.go
│ ├── model.go
│ ├── repository.go
│ └── service.go
├── main.go
└── profile
├── httphandler.go
├── model.go
├── repository.go
└── service.go
Clean Architecture
You have 4 layers of your application or module (depending on how big your codebase is): Domain, Application, Ports, Adapters. In some sources, names may differ.
The src directory is not needed and in fact a lot of public repositories do not use this structure.
There is a few different way of organizing your project. If you plan on having your project used by an other repository, like lib. I would recommend using a cmd struct something like this. This would also be the recommended way of doing it if there would be more then one way of starting the application. (multipliable main.go files)
├──cmd/
| ├──(application name)
| | ├──main.go
└──say/
├──say.go
└──say_test.go
Otherwise for example if it is a standalone application. You can place the main.go in the root of the repository.
bin and pkg you can keep in the root and add this to .gitignore. (assuming you are using git)
The book describes the directory structure after checkout. It would have been helpful if the book included the .git directory.
$GOPATH/src is required for imports to work.
├──src/
| ├──.git
| | ├──...
| ├──main.go
| ├──say/
| | ├──say.go
| | ├──say_test.go
├──bin/
| ├──say
└──pkg/
└──linux_amd64/
└──say.a
In practice, main.go would actually be in a path that reflects the remote git repository, for instance
.
├── bin
│   └── say
├── pkg
│   └── linux_amd64
│   └── github.com
│      └── pschultz
│      └── hello-world
│   └── say.a
└── src
   └── github.com
   └── pschultz
   └── hello-world
   ├── .git
   │ └── ...
   ├── main.go
   └── say
   ├── say.go
   └── say_test.go

Resources