I feel like this is probably an over-asked question on SO yet here it is again. I'm finding this simple task incredibly tedious in Go. Note that I have GO11MODULES set to ON, I'm not sure if this effects the whole package system (it shouldn't is what I'm assuming).
I have a package called "users" which contains a compiled Protocol Buffer (from a .proto file). I want to store it alongside a number of other definitions in a folder called protos. So that my structure looks like so:
- main.go
- protos
- users.go
- users.proto
- analytics.go
- analytics.proto
Pretty simple structure. Within the users.go file I'm defining package protos. Within main.go I'd like to import users "protos/users". When I do so I get this: build command-line-arguments: cannot load protos/users: cannot find module providing package protos/users.
I've followed (I think) other sample code that has done the same thing. Note that the folder structure is within $GOPATH/src/myapi.
Why is this more complicated than its proving to be?
If you are using package protos, then the package is protos. protos/users does not exist. Packages and package imports are directory-level, not file-level. The full import statement depends on the module declaration in your go.mod file, which defines the root of imports. E.g., if your go.mod begins with
module github.com/me/myapp
Then your import would be
import "github.com/me/myapp/protos"
This answer assumes that GO111MODULE is set to on. The question shows that you are setting GO11MODULES. I assume that this is a typo. Fix it if it's not a typoo.
Add file go.mod in same directory as main.go with the following contents:
module myapi
Change main to import "myapi/protos" instead of "protos/users"
Related
Not so familiar with Golang, it's probably a stupid mistake I made... But still, I can't for the life of me figure it out.
So, I got a proto3 file (let's call it file.proto), whose header is as follows:
syntax = "proto3";
package [package_name];
option go_package = "github.com/[user]/[repository]";
And I use protoc:
protoc --go_out=$GOPATH/src --go-grpc_out=$GOPATH/src file.proto
So far so good, I end up with two generated files (file.pb.go and file_grpc.pb.go) inside /go/src/github.com/[user]/[repository]/, and they are defined inside the package [package_name].
Then, the code I'm trying to build has the following import:
import (
"github.com/[user]/[repository]/[package_name]"
)
And I naively thought it would work. However, it produces the following error when running go mod tidy:
go: downloading github.com/[user]/[repository] v0.0.0-20211105185458-d7aab96b7629
go: finding module for package github.com/[user]/[repository]/[package_name]
example/xxx imports
github.com/[user]/[repository]/[package_name]: module github.com/[user]/[repository]#latest found (v0.0.0-20211105185458-d7aab96b7629), but does not contain package github.com/[user]/[repository]/[package_name]
Any idea what I'm doing wrong here? Go version is go1.19 linux/amd64 within Docker (golang:1.19-alpine).
Note: I also tried to only import github.com/[user]/[repository], same issue obviously.
UPDATE:
OK so what I do is that I get the proto file from the git repository that only contains the proto file:
wget https://raw.githubusercontent.com/[user]/[repository]/file.proto
Then I generate go files from that file with protoc:
protoc --go_out=. --go-grpc_out=. file.proto
Right now, in current directory, it looks like:
- directory
| - process.go
| - file.proto
| - github.com
| - [user]
| - [repository]
| - file.pb.go
| - file_grpc.pb.go
In that same directory, I run:
go mod init xxx
go mod tidy
CGO_ENABLED=0 go build process.go
The import directive in process.go is as follows:
import (
"xxx/github.com/[user]/[repository]"
)
Now it looks like it finds it, but still getting a gRPC error, which is weird because nothing changed. I still have to figure out if it comes from the issue above or not. Thanks!
Your question is really a number of questions in one; I'll try to provide some info that will help. The initial issue you had was because
At least one file with the .go extension must be present in a directory for it to be considered a package.
This makes sense because importing github.com/[user]/[repository] would be fairly pointless if that repository does not contain any .go files (i.e. the go compiler could not really do anything with the files).
Your options are:
Copy the output from protoc directly into your project folder and change the package declarations to match your package. If you do this there is no need for any imports.
Copy (or set go_out argument to protoc) the output from protoc into a subfolder of your project. The import path will then be the value of the module declaration in your go.mod plus the path from the folder that the go.mod is in (this is what you have done).
Store the files in a repo (on github or somewhere else). This does not need to be the same repo as your .proto files if you "want it to be agnostic" (note that 2 & 3 can be combined if the generated files will only be used within one code base or the repo is accessible to all users).
Option 1 is simple but its often beneficial to keep the generated code separate (makes it clear what you should not edit and improves editor autocomplete etc).
Option 2 is OK (especially if protoc writes the files directly and you set go_package appropriately). However issues may arise when the generated files will be used in multiple modules (e.g. as part of your customers code) and your repo is private. They will need to change go_package before running protoc (or search/replace the package declarations) and importing other .proto files may not work well.
Option 3 is probably the best approach in most situations because this works with the go tooling. You can create github.com/[user]/goproto (or similar) and put all of your generated code in there. To use this your customers just need to import github.com/[user]/goproto (no need to run protoc etc).
Go Modules/package intro
The go spec does not detail the format of import paths, leaving it up to the implementation:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
As you are using go modules (pretty much the default now) the implementations rules for resolving package paths (synonym of import path) can be summarised as:
Each package within a module is a collection of source files in the same directory that are compiled together. A package path is the module path joined with the subdirectory containing the package (relative to the module root). For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html".
So if your "module path" (generally the top line in a go.mod) is set to xxx (go mod init xxx) then you would import the package in subfolder github.com/[user]/[repository] with import xxx/github.com/[user]/[repository] (as you have found). If you got rid of the intervening folders and put the files into the [repository] subfolder (directly off your main folder) then it would be import xxx/[repository]
You will note in the examples above that the module names I used are paths to repo (as opposed to the xxx you used in go mod init xxx). This is intentional because it allows the go tooling to find the package when you import it from a different module. For example if you had used go mod init github.com/[user]/[repository] and option go_package = "github.com/[user]/[repository]/myproto";" then the generated files should go into the myproto folder in your project and you import them with import github.com/[user]/[repository]/myproto.
While you do not have to follow this approach I'd highly recommend it (it will save you from a lot of pain!). It can take a while to understand the go way of doing this, but once you do, it works well and makes it very clear where a package is hosted.
I created the github.com project qjson/qjson-go that contains the package name qjson that you can see here. I named the github project this way because github.com/qjson/ contains other projects for different languages (e.g. qjson-c).
Unfortunately, I get the following error when I try to import the project as github.com/qjson/qjson-go:
$ go mod tidy
go: finding module for package github.com/qjson/qjson-go
go: downloading github.com/qjson/qjson-go v0.0.0-20210128102242-170c47e2db46
github.com/xxx/xxx imports
github.com/qjson/qjson-go: module github.com/qjson/qjson-go#latest found (v0.0.0-20210128102242-170c47e2db46), but does not contain package github.com/qjson/qjson-go
I’m apparently doing it wrong. I understand that due to the import statement we are then expected to use gjson-go as package identifier.
What must I do so that the git project can be named qjson-go and the package qjson ?
I assume that one solution is to create a sub-directory named qjson inside qjson-go and move all package files in it. The user would then import "github.com/qson/qson-go/qjson". Is that correct ? Is there another solution avoiding the stutter ?
This program works as expected:
package main
import (
"fmt"
"github.com/qjson/qjson-go/qjson"
)
func main() {
fmt.Println(qjson.ErrDivisionByZero)
}
The issue is that you are using this file structure:
qjson/engine.go
qjson/errors.go
When you should just be putting them at the top level, like this:
engine.go
errors.go
So you can either fix the directory, and tag a new version, or just leave the
files as is, and change your imports to match what I have above.
I can not find a way to factor out some code from main.go into a local package when using Go modules (go version >= 1.11) outside of $GOPATH.
I am not importing any external dependencies that need to be included into go.mod, I am just trying to organize locally the source code of this Go module.
The file main.go:
package main
// this import does not work
import "./stuff"
func main() {
stuff.PrintBaz()
}
The file ./stuff/bar.go (pretending to be a local package):
package stuff
import "log"
type Bar struct {
Baz int
}
func PrintBaz() {
baz := Bar{42}
log.Printf("Bar struct: %v", baz)
}
The file go.mod (command go mod init foo):
module foo
go 1.12
When executing go run main.go:
If I import "./stuff", then I see build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
If I import "stuff", then I see build command-line-arguments: cannot load stuff: cannot find module providing package stuff.
If I import stuff "./stuff" with a package alias, then I see again: build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
I can not find a way to make local packages work with go modules.
What's wrong with the code above?
How can I import local packages into other Go code inside a project defined with Go modules (file go.mod)?
Module structure
The most common and easiest approach is:
Use a single go.mod per repository, and
Place the single go.mod file in the repository root, and
Use the repository name as the module path declared in the module line in the go.mod
(If you are using a custom import path such as me.io/mymod rather than using a VCS host based import path, then you would use the custom import path instead of the repository name in your go.mod).
For example, if your repo is github.com/my/repo, then you would place a single go.mod in the repo root, with the first line reading module github.com/my/repo. That can be created by cd'ing to the repo root and running go mod init github.com/my/repo. When working with the repo locally, the repo directory can be located wherever is convenient on your filesystem. (If you do not have a repo, see below).
Following this helps you stay on the happy path with modules, and it avoids multiple subtleties.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
There is much more about multi-module repositories in the "Multi-module Repositories" FAQ section on the modules wiki. Those 6 or so FAQs in that section should be read in their entirety by anyone considering veering off the recommendation above.
Arranging packages within a module
Once you have set up your go.mod, you can arrange your packages in directories however you see fit in directories underneath the directory containing the go.mod, as well as in the directory with the go.mod. Three good articles about how to arrange your code in packages:
https://rakyll.org/style-packages/
https://medium.com/#benbjohnson/standard-package-layout-7cdbc8391fc1#.ds38va3pp
https://www.goinggo.net/2017/02/design-philosophy-on-packaging.html
Those articles are classics that pre-date the introduction of modules, but the philosophies in them still apply to how to arrange your packages within a module.
Importing other packages in the same module
When importing another package with modules, you always use the full path including the module path. This is true even when importing another package in the same module. For example, if a module declared its identity in its go.mod as module github.com/my/repo, and you had this organization:
repo/
├── go.mod <<<<< Note go.mod is located in repo root
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
Then pkg1 would import its peer package as import "github.com/my/repo/pkg2". Note that you cannot use relative import paths like import "../pkg2" or import "./subpkg". (This is part of what OP hit above with import "./stuff").
Modules vs. repositories vs. packages vs. import paths
A Go module is a collection of related Go packages that are versioned together as a single unit. Modules record precise dependency requirements and create reproducible builds.
Summarizing the relationship between repositories, modules, and packages:
A repository contains one or more Go modules (most often exactly one module in the repository root).
Each module contains one or more Go packages.
Each package consists of one or more Go source files that all reside in a single directory.
Each Go source code file:
declares its own package with a package foo statement, which must be consistent for all Go files in a single package directory.
automatically has access to other Go source code in the same package, without explicitly importing its own package.
imports code from another package via an import path supplied in an import statement such as import "github.com/my/repo/pkg1". The import path always starts with the module path of that package, regardless of whether that package is in the same module or a different module.
A single module can be used to build multiple executable binaries, but the func main() {...} entry point for each binary must be in its own package main, with each package main in a distinct directory.
Local-only modules
You can create, build and run modules without ever publishing them or placing them on a VCS server.
A module is defined by a tree of Go source files with a go.mod file in the tree's root directory, so you can pick a directory on your local filesystem to serve as the root of your module and place your go.mod there. Everything described above still applies, except there does not need to be a repo.
For example:
some-directory/
├── go.mod <<<< go.mod at module root declaring 'module example.com/foo'
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
If the module path in the first line of that go.mod reads module example.com/foo, then pkg1 would import its peer package as import "example.com/foo/pkg2".
Of course, you can start with a local-only module without a repo, then later start tracking it in a local repo or publish it to a VCS server.
Even for local-only modules, it is a good idea to give the module a globally unique name, such as go mod init my.org/foo, or go mod init github.com/some/repo as if it will be published someday, even if that repo does not yet exist. (If you are in a hurry, it is possible to use a shorter module name without a dot or leading hostname component, such as go mod init temp or go mod init m, but it won’t be fetchable by the go command, and you run the risk of colliding with a current or future standard library package or otherwise seeing confusing error messages later).
If you have multiple local modules, you can point them at each other via replace directives or via the Go 1.18+ workspace feature, though those are more complex workflows than placing everything for a given project in a single module.
Next steps
If you are new to Go modules, it is worthwhile to go through the official "Create a Go Module" tutorial from the Go project. This will likely save you time overall.
If you are stuck right now with an error message related to modules or import paths, there's a very good chance the error will make much more sense after completing the tutorial.
First you have to choose a name for your project and write it to go.mod file. This name belongs to root directory of the project. Each new package you create must be located inside its own subdirectory and its name should match directory name.
go.mod:
module myprojectname
or (preferred way, see #thepudd's answer for details)
module github.com/myname/myproject
Then import your project's packages like:
import myprojectname/stuff
or
import github.com/myname/myproject/stuff
Files of package stuff should be located inside project's stuff directory. You name these files as you like.
Also it's possible to create deeper project structure. For instance, you decided to separate source code files from other ones (like app configs, docker files, static files, etc...). Let's move stuff directory inside pkg, every go file inside pkg/stuff still have stuff package name. To import stuff package just write:
import myprojectname/pkg/stuff
Nothing stops you from creating more levels in the hierarchy like github.com/myuser/myproject/pkg/db/provider/postgresql, where:
github.com/myuser/myproject - project name.
postgresql - package name.
pkg/db/provider/postgresql - path to the package relative to project's root.
You can read more about go modules here: https://github.com/golang/go/wiki/Modules
Check out this repository to get useful information about various patterns used in project organizing: https://github.com/golang-standards/project-layout If you go inside pkg directory you will find out which open source projects use pkg directory in their structure.
I'm trying out Go's new modules system and am having trouble accessing local packages. The following project is in a folder on my desktop outside my gopath.
My project structure looks like:
/
- /platform
- platform.go
- main.go
- go.mod
// platform.go
package platform
import "fmt"
func Print() {
fmt.Println("Hi")
}
// main.go
package main
import "platform"
func main() {
platform.Print()
}
go build main.go tells me
cannot find module for path platform
Let me define this first modules are collections of packages. In Go 11, I use go modules like the following:
If both packages are in the same project, you could just do the following:
In go.mod:
module github.com/userName/moduleName
and inside your main.go
import "github.com/userName/moduleName/platform"
However, if they are separate modules, i.e different physical paths and you still want to import local packages without publishing this remotely to github for example, you could achieve this by using replace directive.
Given the module name github.com/otherModule and platform, as you've called it, is the only package inside there. In your main module's go.mod add the following lines:
module github.com/userName/mainModule
require "github.com/userName/otherModule" v0.0.0
replace "github.com/userName/otherModule" v0.0.0 => "local physical path to the otherModule"
Note: The path should point to the root directory of the module, and can be absolute or relative.
Inside main.go, to import a specific package like platform from otherModule:
import "github.com/userName/otherModule/platform"
Here's a gentle introduction to Golang Modules
I would strongly suggest you to use go toolchain which takes care of these issues out of the box. Visual Studio Code with vscode-go plugin is really useful.
Problem here is that Go requires relative paths with respect to your $GOPATH/src or module in import statement. Depending on where you are in your GOPATH, import path should include that as well. In this case, import statement must include go module path in go.mod
GOPATH
Assume your project resides here:
$GOPATH/src/github.com/myuser/myproject
Your import path should be:
import "github.com/myuser/myproject/platform"
VGO
Assume your go.mod file is:
module example.com/myuser/myproject
Your import path should be:
import "example.com/myuser/myproject/platform"
As someone new to go I didn't immediately understand the accepted answer – which is great, by the way. Here's a shorter answer for those very new people!
In go modules/packages are expressed as urls that you import in your code:
import your.org/internal/fancy_module
But wait! My code isn't at a url, what's happening??
This is the cleverness of go. You pretend there's a url even when there isn't one. Because:
This makes including easier as no matter where your file is located the import uses the same url (the code stays the same even if the files move!)
You can have packages that having naming conflicts. So google.com/go-api/user doesn't conflict with the definitions at your.org/internal/user
Someday you might publish a url on GitHub and all the code will just work
That's all great Evan, but how do I import a relative path?
Good question! You can import by changing your go.mod file to have this line:
module fancy_module
go 1.16
replace your.org/fancy_module => ../path/to/fancy_module
Given the Golang Project structure
/
- /platform
- platform.go
- main.go
- go.mod
To access the methods or structs...etc (which are public) from local packages of /platform is simple, shown below
// main.go
package main
import (
p "./platform"
)
func main() {
p.Print()
}
this should work
In the process of learning go I was playing around with making my own libraries. Here is what I did: in my $GOPATH/src I have two folders: mylibs and test. The test folder has a file called test.go which contains
package test
import "mylibs/hi/saysHi"
func main() {
saysHi.SayHi()
}
The mylibs folder contains another folder called hi, which has a file called saysHi.go containing:
package saysHi
import "fmt"
func SayHi() {
fmt.Printf("Hi\n")
}
So the directory structure looks like this:
GOPATH/src
test
test.go
mylibs
hi
saysHi.go
The problem is that when I try to compile test it complains saying
cannot find package "mylibs/hi/saysHi" in any of:
[...]
$GOPATH/src/mylibs/hi/saysHi (from $GOPATH)
I have deliberately made the directory structure deeper than necessary. If I make a simpler directory structure where I place saysHi.go in $GOPATH/saysHi/saysHi.go then it works.
But I don't see a reason for why this wouldn't work. Any ideas?
Generally speaking, your directory name should match the package name. So if you define
package saysHi
and want to import it with
import "mylibs/hi/saysHi"
you should place it in a structure like this:
mylibs
hi
saysHi
saysHi.go
The name of the .go file(s) inside the package makes no difference to the import path, so you could call the .go file anything you like.
To explain it a bit further, the import path you use should be the name of the directory containing the package. But, if you define a different package name inside that directory, you should use that name to access the package inside the code. This can be confusing, so it's best to avoid it until you understand where it's best used (hint: package versioning).
It gets confusing, so for example, if you had your package in the path
mylibs
hi
saysHi.go
And inside saysHi.go defined,
package saysHi
Then in test.go you will import it with
import "mylibs/hi"
And use it with
saysHi.SayHi()
Notice how you import it with the final directory being hi, but use it with the name saysHi.
Final note
Just in case you didn't know the following: your test file is called test.go, and that's fine, if it's just as an example, and not an actual test file for saysHi.go. But if it is/were a file containing tests for saysHi.go, then the accepted Go standard is to name the file saysHi_test.go and place it inside the same package alongside saysHi.go.
One more final note
I mentioned how you are allowed to choose a different package name from the directory name. But there is actually a way to write the code so that it's less confusing:
import (
saysHi "mylibs/hi"
)
Would import it from the mylibs/hi directory, and make a note of the fact that it should be used with saysHi, so readers of your code understand that without having to go look at the mylibs/hi code.