I have a Golang project with one git submodule like follows:
Project
go.mod
main.go
config.go
go-blockchain/ (git Submodule)
go.mod
crypto/
encode.go (This has cgo code in it)
crypto.go
main.go
When I run go build at the top level, I get errors like:
# github.com/blockchain/go-blockchain/crypto
/Users/niceguy/.go/pkg/mod/github.com/blockchain/go-blockchain#v0.0.0-20200320145517-cea8009ba7ee/crypto/encode.go:86:2: could not determine kind of name for C.crypto_encode_pair
/Users/niceguy/.go/pkg/mod/github.com/blockchain/go-blockchain#v0.0.0-20200320145517-cea8009ba7ee/crypto/encode.go:80:2: could not determine kind of name for C.crypto_encode_from_seed
/Users/niceguy/.go/pkg/mod/github.com/blockchain/go-blockchain#v0.0.0-20200320145517-cea8009ba7ee/crypto/encode.go:115:9: could not determine kind of name for C.crypto_encode_proof_to_hash
The problem appears to be that I'm favoring the cache over the CGO libraries that are built and deposited back in the go-blockchain folder.
Suggestions?
Related
I ran into a repo that seems to be a Go module, but there's no go.mod file in it: github.com/confluentinc/confluent-kafka-go.
Is it ok for a go module to have no go.mod file with dependencies, or the authors of that library just didn't migrate to modules yet?
Dependency modules do not need to have explicit go.mod files.
The “main module” in module mode — that is, the module containing the working directory for the go command — must have a go.mod file, so that the go command can figure out the import paths for the packages within that module (based on its module path), and so that it has a place to record its dependencies once resolved.
In addition, any modules slotted in using replace directives must have go.mod files (in order to reduce confusion due to typos or other errors in replacement paths).
However, in general a module that lacks an explicit go.mod file is valid and fine to use. Its effective module path is the path by which it was required, which can be a bit confusing if the same repository ends up in use via multiple paths. Since a module with no go.mod file necessarily doesn't specify its own dependencies, consumers of that module will have to fill in those dependencies themselves (go mod tidy will mark them as // indirect in the consumer's go.mod file).
SHORT SUMMARY OF THE DISCUSSION:
The answer is "No"!
This project contains a set of go packages, but it is not a Go module as it doesn't contain go.mod file (although, it used to be a multi-module repo (Go) previously).
go get can run in both ways: module-aware mode and legacy GOPATH mode (as of Go 1.16).
To read more about this, refer to docs by using the go command:
$ go help gopath-get
and
$ go help module-get
It'd tell about how go get works in both cases.
Also, I noticed that it can download any repository and would treat it as a Go package, even if it contains an arbitrary Python project.
I did a simple test to demonstrate the same:
$ go get github.com/mongoengine/mongoengine
And it surprisingly worked.
Modules are defined by their go.mod file. Without a go.mod file, it is not a module.
See this from the Go Modules Reference
A module is a collection of packages that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or from module proxy servers.
A module is identified by a module path, which is declared in a go.mod file, together with information about the module's dependencies. The module root directory is the directory that contains the go.mod file.
And
A module is defined by a UTF-8 encoded text file named go.mod in its root directory.
I am confused about how to layout my Go project and then how to build it. I am reasonably new to Go and believe there was a time <1.13 where the GOPATH was important. I am using 1.14, so I believe I do not have to care about that and GOPATH is not set. I do not (for the moment) host my code on GitHub (which is something various articles assume). I have read a number of things, but it all leaves me more confused:
https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies
https://www.wolfe.id.au/2020/03/10/starting-a-go-project/
https://talks.golang.org/2014/organizeio.slide#1
I have laid out my project according to this: https://github.com/golang-standards/project-layout. (Except this project seems to use a Makefile. I do not want to write a Makefile. I believe this should all work without a Makefile. I really do not want to write a Makefile.)
Here is the structure:
/src
/cmd
main.go
/internal
helper.go
go.mod
go.mod looks like this:
module mycompany/mymodule
go 1.14
service.go looks as follows
package main // this has to be called 'main'
import (
"mycompany/mymodule/internal/helper"
)
func main () {
helper.greet("Jenny")
}
So, if I am in /src and I run build cmd/service.go I get
cmd/service.go:4:2: package mycompany/mymodule/internal/helper is not in GOROOT (/usr/local/Cellar/go/1.14.5/libexec/src/mycompany/mymodule/internal/helper)
Do I have to compile helper.go first? Manually? Surely not. go build surely should be able to build my entire project, right? Including all the dependencies. Could someone please tell me what I am missing?
When you run go build without specifying file path, it will look for the main function inside any .go files at current directory to start. It doesn't build entire project. But it wouldn't be a problem in this case.
You're importing wrong package. The package name to import must be mycompany/mymodule/internal not mycompany/mymodule/internal/helper.
And to invoke a function inside another package, it must be exposed to outside.
So you have to declare the function greet() as Greet()
I have two git repositories:
common
this repository contains protobuf files (.proto), these files should be fetched via git submodule init from other vendor repository. There are also go files with directive: //go:generate protoc --proto_path=a/xxx --go_out=. --go_opt=paths=source_relative a/xxx/b.proto
main
this repository contains main project, which imports the package from common. Then I call git submodule update --init and the go generate. After generate i can see the log: go: finding github.xxx.com/xxx latest
but the main go file shows some problems:
build github.xxx.com/main: cannot load github.xxx.com/xxx/proto: module github.xxx.com/common#latest found, but does not contains package github.xxx.com/xxx/proto
It seems that the go generate command doesn't generate proto files. I think if I import something then these files are somewhere in cache. I'm not sure if my approach is possible.
Could you tell me if this solution is feasible or what I should change.
ok, my solution is project layout like this: https://github.com/golang-standards/project-layout
there I have more main.go files inside cmd folder. In this way I have only one module and inside the module I can have different applicaions. I think it's very cool solution.
I can not find a way to factor out some code from main.go into a local package when using Go modules (go version >= 1.11) outside of $GOPATH.
I am not importing any external dependencies that need to be included into go.mod, I am just trying to organize locally the source code of this Go module.
The file main.go:
package main
// this import does not work
import "./stuff"
func main() {
stuff.PrintBaz()
}
The file ./stuff/bar.go (pretending to be a local package):
package stuff
import "log"
type Bar struct {
Baz int
}
func PrintBaz() {
baz := Bar{42}
log.Printf("Bar struct: %v", baz)
}
The file go.mod (command go mod init foo):
module foo
go 1.12
When executing go run main.go:
If I import "./stuff", then I see build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
If I import "stuff", then I see build command-line-arguments: cannot load stuff: cannot find module providing package stuff.
If I import stuff "./stuff" with a package alias, then I see again: build command-line-arguments: cannot find module for path _/home/<PATH_TO>/fooprj/stuff.
I can not find a way to make local packages work with go modules.
What's wrong with the code above?
How can I import local packages into other Go code inside a project defined with Go modules (file go.mod)?
Module structure
The most common and easiest approach is:
Use a single go.mod per repository, and
Place the single go.mod file in the repository root, and
Use the repository name as the module path declared in the module line in the go.mod
(If you are using a custom import path such as me.io/mymod rather than using a VCS host based import path, then you would use the custom import path instead of the repository name in your go.mod).
For example, if your repo is github.com/my/repo, then you would place a single go.mod in the repo root, with the first line reading module github.com/my/repo. That can be created by cd'ing to the repo root and running go mod init github.com/my/repo. When working with the repo locally, the repo directory can be located wherever is convenient on your filesystem. (If you do not have a repo, see below).
Following this helps you stay on the happy path with modules, and it avoids multiple subtleties.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
There is much more about multi-module repositories in the "Multi-module Repositories" FAQ section on the modules wiki. Those 6 or so FAQs in that section should be read in their entirety by anyone considering veering off the recommendation above.
Arranging packages within a module
Once you have set up your go.mod, you can arrange your packages in directories however you see fit in directories underneath the directory containing the go.mod, as well as in the directory with the go.mod. Three good articles about how to arrange your code in packages:
https://rakyll.org/style-packages/
https://medium.com/#benbjohnson/standard-package-layout-7cdbc8391fc1#.ds38va3pp
https://www.goinggo.net/2017/02/design-philosophy-on-packaging.html
Those articles are classics that pre-date the introduction of modules, but the philosophies in them still apply to how to arrange your packages within a module.
Importing other packages in the same module
When importing another package with modules, you always use the full path including the module path. This is true even when importing another package in the same module. For example, if a module declared its identity in its go.mod as module github.com/my/repo, and you had this organization:
repo/
├── go.mod <<<<< Note go.mod is located in repo root
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
Then pkg1 would import its peer package as import "github.com/my/repo/pkg2". Note that you cannot use relative import paths like import "../pkg2" or import "./subpkg". (This is part of what OP hit above with import "./stuff").
Modules vs. repositories vs. packages vs. import paths
A Go module is a collection of related Go packages that are versioned together as a single unit. Modules record precise dependency requirements and create reproducible builds.
Summarizing the relationship between repositories, modules, and packages:
A repository contains one or more Go modules (most often exactly one module in the repository root).
Each module contains one or more Go packages.
Each package consists of one or more Go source files that all reside in a single directory.
Each Go source code file:
declares its own package with a package foo statement, which must be consistent for all Go files in a single package directory.
automatically has access to other Go source code in the same package, without explicitly importing its own package.
imports code from another package via an import path supplied in an import statement such as import "github.com/my/repo/pkg1". The import path always starts with the module path of that package, regardless of whether that package is in the same module or a different module.
A single module can be used to build multiple executable binaries, but the func main() {...} entry point for each binary must be in its own package main, with each package main in a distinct directory.
Local-only modules
You can create, build and run modules without ever publishing them or placing them on a VCS server.
A module is defined by a tree of Go source files with a go.mod file in the tree's root directory, so you can pick a directory on your local filesystem to serve as the root of your module and place your go.mod there. Everything described above still applies, except there does not need to be a repo.
For example:
some-directory/
├── go.mod <<<< go.mod at module root declaring 'module example.com/foo'
├── pkg1
│ └── pkg1.go
└── pkg2
└── pkg2.go
If the module path in the first line of that go.mod reads module example.com/foo, then pkg1 would import its peer package as import "example.com/foo/pkg2".
Of course, you can start with a local-only module without a repo, then later start tracking it in a local repo or publish it to a VCS server.
Even for local-only modules, it is a good idea to give the module a globally unique name, such as go mod init my.org/foo, or go mod init github.com/some/repo as if it will be published someday, even if that repo does not yet exist. (If you are in a hurry, it is possible to use a shorter module name without a dot or leading hostname component, such as go mod init temp or go mod init m, but it won’t be fetchable by the go command, and you run the risk of colliding with a current or future standard library package or otherwise seeing confusing error messages later).
If you have multiple local modules, you can point them at each other via replace directives or via the Go 1.18+ workspace feature, though those are more complex workflows than placing everything for a given project in a single module.
Next steps
If you are new to Go modules, it is worthwhile to go through the official "Create a Go Module" tutorial from the Go project. This will likely save you time overall.
If you are stuck right now with an error message related to modules or import paths, there's a very good chance the error will make much more sense after completing the tutorial.
First you have to choose a name for your project and write it to go.mod file. This name belongs to root directory of the project. Each new package you create must be located inside its own subdirectory and its name should match directory name.
go.mod:
module myprojectname
or (preferred way, see #thepudd's answer for details)
module github.com/myname/myproject
Then import your project's packages like:
import myprojectname/stuff
or
import github.com/myname/myproject/stuff
Files of package stuff should be located inside project's stuff directory. You name these files as you like.
Also it's possible to create deeper project structure. For instance, you decided to separate source code files from other ones (like app configs, docker files, static files, etc...). Let's move stuff directory inside pkg, every go file inside pkg/stuff still have stuff package name. To import stuff package just write:
import myprojectname/pkg/stuff
Nothing stops you from creating more levels in the hierarchy like github.com/myuser/myproject/pkg/db/provider/postgresql, where:
github.com/myuser/myproject - project name.
postgresql - package name.
pkg/db/provider/postgresql - path to the package relative to project's root.
You can read more about go modules here: https://github.com/golang/go/wiki/Modules
Check out this repository to get useful information about various patterns used in project organizing: https://github.com/golang-standards/project-layout If you go inside pkg directory you will find out which open source projects use pkg directory in their structure.
Following the release of Go 1.11, I have been trying to move my repositories to Go modules, by adding a go.mod file at their root.
One of my root libraries my.host/root is in its version 17.0.1, so I wrote in its go.mod file:
module my.host/root/v17
I tagged that version v17.0.1 as documented in the Go modules manual.
When I try to make a new Go project that uses my root library, like:
package main
import root "my.host/root/v17"
func main() {
root.DoSomething()
}
And try to compile it, I get the following error:
go: my.host/root#v0.0.0-20180828034419-6bc78016491a: go.mod has post-v0 module path "my.host/root/v17" at revision 6bc78016491a
I am at loss figuring out why this happens. I explicitly added v17.0.1 in the go.mod file, yet every attempt at go build replaces the entry with a v0.0.0-20180828034419-6bc78016491a version which then fails because at that commit, the go.mod file module entry of my root library indeed ends with a v17, as it should.
For the record, this commit is the same as the tagged v17.0.1 version.
What am I doing wrong here? How can I debug this situation?
I had make two mistakes:
My initial v17.0.0 tag would point to a commit where go.mod did not contain the v17 import path suffix. As a result, it seems Go tooling considers the whole v17 major version as a v0/v1 instead, even if later v17 tags point to a commit with a correct go.mod directive, hence the commit ID "translation".
In my dependent projects, in the go.mod file, I mistakenly specified
require my.host/root v17.0.1 instead of
require my.host/root/v17 v17.0.1.
After fixing both those issues, everything seems back to normal and it works perfectly. I wish the documentation had been clearer about this but I guess this is a good opportunity to make a contribution!
The error I got was: github.com/emicklei/go-restful#v0.0.0-20180531035034-3658237ded10: go.mod has post-v0 module path "github.com/emicklei/go-restful/v2" at revision 3658237ded10
Appending github.com/emicklei/go-restful with v2 like so: github.com/emicklei/go-restful/v2 in my go.mod file fixed it for me.