I have built a go package (https://github.com/emicklei/go-restful) and I am now trying to reference the output of that package build from my own workspace.
$GOPATH
|--src
| |-- {my source}
|--pkg
| |-- {package binaries /linux_amd64/github.com/emicklei/go-restful.a}
|--bin
|-- {my output}
but this fails to compile stating can not find package. go build is only searching the src folder as AFAIK and all the solutions I can find around this involve having the package source code, but surely it's possible to reference package binaries?
Turns out it's not possible to reference a package binary, in order to reference a package the source code is required: http://zduck.com/2014/go-and-package-versioning/
============ EDIT ==============
Turns out you can hack it so that the compiler will use the package output by placing a dummy source file in the expected location with a last modified older than the package output. (see Volker answer to original question).
Related
Not so familiar with Golang, it's probably a stupid mistake I made... But still, I can't for the life of me figure it out.
So, I got a proto3 file (let's call it file.proto), whose header is as follows:
syntax = "proto3";
package [package_name];
option go_package = "github.com/[user]/[repository]";
And I use protoc:
protoc --go_out=$GOPATH/src --go-grpc_out=$GOPATH/src file.proto
So far so good, I end up with two generated files (file.pb.go and file_grpc.pb.go) inside /go/src/github.com/[user]/[repository]/, and they are defined inside the package [package_name].
Then, the code I'm trying to build has the following import:
import (
"github.com/[user]/[repository]/[package_name]"
)
And I naively thought it would work. However, it produces the following error when running go mod tidy:
go: downloading github.com/[user]/[repository] v0.0.0-20211105185458-d7aab96b7629
go: finding module for package github.com/[user]/[repository]/[package_name]
example/xxx imports
github.com/[user]/[repository]/[package_name]: module github.com/[user]/[repository]#latest found (v0.0.0-20211105185458-d7aab96b7629), but does not contain package github.com/[user]/[repository]/[package_name]
Any idea what I'm doing wrong here? Go version is go1.19 linux/amd64 within Docker (golang:1.19-alpine).
Note: I also tried to only import github.com/[user]/[repository], same issue obviously.
UPDATE:
OK so what I do is that I get the proto file from the git repository that only contains the proto file:
wget https://raw.githubusercontent.com/[user]/[repository]/file.proto
Then I generate go files from that file with protoc:
protoc --go_out=. --go-grpc_out=. file.proto
Right now, in current directory, it looks like:
- directory
| - process.go
| - file.proto
| - github.com
| - [user]
| - [repository]
| - file.pb.go
| - file_grpc.pb.go
In that same directory, I run:
go mod init xxx
go mod tidy
CGO_ENABLED=0 go build process.go
The import directive in process.go is as follows:
import (
"xxx/github.com/[user]/[repository]"
)
Now it looks like it finds it, but still getting a gRPC error, which is weird because nothing changed. I still have to figure out if it comes from the issue above or not. Thanks!
Your question is really a number of questions in one; I'll try to provide some info that will help. The initial issue you had was because
At least one file with the .go extension must be present in a directory for it to be considered a package.
This makes sense because importing github.com/[user]/[repository] would be fairly pointless if that repository does not contain any .go files (i.e. the go compiler could not really do anything with the files).
Your options are:
Copy the output from protoc directly into your project folder and change the package declarations to match your package. If you do this there is no need for any imports.
Copy (or set go_out argument to protoc) the output from protoc into a subfolder of your project. The import path will then be the value of the module declaration in your go.mod plus the path from the folder that the go.mod is in (this is what you have done).
Store the files in a repo (on github or somewhere else). This does not need to be the same repo as your .proto files if you "want it to be agnostic" (note that 2 & 3 can be combined if the generated files will only be used within one code base or the repo is accessible to all users).
Option 1 is simple but its often beneficial to keep the generated code separate (makes it clear what you should not edit and improves editor autocomplete etc).
Option 2 is OK (especially if protoc writes the files directly and you set go_package appropriately). However issues may arise when the generated files will be used in multiple modules (e.g. as part of your customers code) and your repo is private. They will need to change go_package before running protoc (or search/replace the package declarations) and importing other .proto files may not work well.
Option 3 is probably the best approach in most situations because this works with the go tooling. You can create github.com/[user]/goproto (or similar) and put all of your generated code in there. To use this your customers just need to import github.com/[user]/goproto (no need to run protoc etc).
Go Modules/package intro
The go spec does not detail the format of import paths, leaving it up to the implementation:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
As you are using go modules (pretty much the default now) the implementations rules for resolving package paths (synonym of import path) can be summarised as:
Each package within a module is a collection of source files in the same directory that are compiled together. A package path is the module path joined with the subdirectory containing the package (relative to the module root). For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html".
So if your "module path" (generally the top line in a go.mod) is set to xxx (go mod init xxx) then you would import the package in subfolder github.com/[user]/[repository] with import xxx/github.com/[user]/[repository] (as you have found). If you got rid of the intervening folders and put the files into the [repository] subfolder (directly off your main folder) then it would be import xxx/[repository]
You will note in the examples above that the module names I used are paths to repo (as opposed to the xxx you used in go mod init xxx). This is intentional because it allows the go tooling to find the package when you import it from a different module. For example if you had used go mod init github.com/[user]/[repository] and option go_package = "github.com/[user]/[repository]/myproto";" then the generated files should go into the myproto folder in your project and you import them with import github.com/[user]/[repository]/myproto.
While you do not have to follow this approach I'd highly recommend it (it will save you from a lot of pain!). It can take a while to understand the go way of doing this, but once you do, it works well and makes it very clear where a package is hosted.
My folder structure looks something like this... (say my git repo name is demorepo)
demorepo
|--directory1
|-- (no go.mod at this level)
|--module1
|--package1 --------->--------------->--------------------->----|
|--go.mod (github.com/demorepo/directory1/module1) |
|--go.sum |
|--module2 |
|--module3 |
|--directory2 |
|-- (no go.mod at this level) |
|--newmodule ------<------------<------------------<----------------|
Now, I want to use a function defined in "package1" in my "newmodule"
When I hit
go get <repo_address>/directort1/module1/package1 at "new_module"
it says ....
github.com/<repo>#upgrade found (v0.0.0-20211215055943-92e412ad4a12), but does not contain package github.com/<repo>/directory1/module1/package1
There is a proposal for a Go Workspace File for Go 1.18 which should simplify this task.
Meanwhile, you can use the replace directive in your go.mod file to refer to a module located on a local filesystem.
demorepo/directory1/module1/go.mod:
module github.com/<repo>/directory1/module1
demorepo/directory2/newmodule/go.mod:
module github.com/<repo>/directory2/newmodule
replace github.com/<repo>/directory1/module1 => ../../directory1/module1
Now you can import github.com/<repo>/directory1/module1/package1 in newmodule normally and it'll refer to the local module1.
You might not want the replace directive in the go.mod file itself, and instead make a copy of it e.g. go.mod.local and use it when building your project: go build -modfile go.mod.local . (can also add *.local to .gitignore).
Approach :
If you want to use module1 inside newModule then you should make one new repo for module1 put your logic there push it in github. please make sure you should use appropriate version for the library.
import it as a library and it will work.
Also refer the official docs of module dependency and also check root level dependency.
Official docs : https://go.dev/blog/using-go-modules
I'm trying to understand how to organize my golang project using go1.11 modules. I tried several options, but none of them worked.
I have some code in the main package under the application folder and a local package that the main package uses.
$GOPATH
+ src
+ application/
+ main/
+ main.go
+ otherFileUnderMainPackage.go
+ aLocalPackage/
+ someCode.go
+ someCode_test.go
+ someMoreCode.go
+ someMoreCode_test.go
Files in the main package, imports ../aLocalPackage. When I compile by go build main/*.go it's working.
Then, I ran go mod init application: V.0.9.9 and got the go.mod file, but the build always fails. I always get error about not finding the local package: build application:V0.9.9/main: cannot find module for path _/.../src/application/aLocalPackage. I also tried to place the local package right under src/, place it under main/ etc. but none of these methods worked for me.
What is the way to use modules and local packages?
Thanks.
Relative import paths are not supported in module mode. You will need to update your import statements to use a full (absolute) import path.
You should also choose a module name other than application. Your module path should generally begin with a URL prefix that you control — either your own domain name, or a unique path such as github.com/$USER/$REPO.
I had some problems working with local packages myself.
There are two tricks to make it work:
you run "go build" in the package directory
This compiles the package and places it in the build cache.
This link about code organisation in go explains more.
You can identify where the cache is using:
>go env GOCACHE
/home/<user>/.cache/go-build
Import using a path relative to the project
I puzzled loads over what the correct import path was and finally discovered that go doc or go list will tell you.
>go doc
package docs // import "tools/src/hello/docs"
>go list
tools/src/hello/docs
For example. I have a hello world API project and was using swaggo to generate documentation which it does in a docs sub-directory.
To use it I add an import:
_ "tools/src/hello/docs"
For my case the _ is important as docs is not used directly but we its init() function to be invoked.
Now in hello/main.go I can add "tools/src/hello/docs" and it will import the correct package.
The path is relative to the location of go.mod if you have one.
I have tools/ here as I have a go.mod declaring "modules tools".
Modules are a different kettle of fish - see https://github.com/golang/go/wiki/Modules.
Recent versions of go (1.11 and later) can create a go.mod file which you may use to fix the version of a module that is used and avoid go's crazy default behaviour of just downloading the latest version of any package you import.
I have written a blogpost on how to start your first Go project using modules.
https://marcofranssen.nl/start-on-your-first-golang-project/
In general it boils down to just create a new folder somewhere on your system (doesn't have to be in GOPATH).
mkdir my-project
cd my-project
go mod init github.com/you-user/my-project
This will create the go.mod file. Now you can simply create your project layout and start building whatever you like.
Maybe one of my other blogs can inspire you a bit more on how to do things.
https://marcofranssen.nl/categories/software-development/golang/
How to setup project structure to produce library and cli with same name in
single repository?
Suppose my project name is project. I want to make it importable with name
project and have executable binary with name project when installed with
go get. My setup currently is like this:
host.com/project/
project/
main.go
core/
project.go
Then, when installed with:
go get host.com/project/project
it installs project as executable which pulls core as dependency. In
core/project.go file, the package has this:
package project
The problem is it is imported with:
import (
"host.com/project/core"
)
And it exports project as name space not core which violates go's convention.
How can I do this?
This isn't a convention, and you may do however you like, but it's a common practice to have a cmd folder in the project root, and all the executables go under that, each in is own folder.
So a good solution in your case could be simply:
host.com/project/
cmd/
project/
project.go
filea.go
fileb.go
In filea.go and fileb.go the package declaration should be:
package project
In cmd/project/project.go package and import declaration should be:
package main
import "host.com/project"
Of course if your project is more complex, you may create further packages under the project root, and it may have multiple commands (multiple main packages), e.g.:
host.com/project/
cmd/
project/
project.go
prjtool/
prjtool.go
packagex/
x.go
packagey/
y.go
filea.go
fileb.go
An example following this layout is the very popular Go Delve debugger (4.5k stars currently), available at https://github.com/derekparker/delve.
A very complex repository example is the golang.org/x/tools package which is a collection of multiple, essential Go tools, available at https://github.com/golang/tools/. Its cmd folder contains more than 20 subfolders (commands), many of the tools (main packages) even consist of multiple .go files (but each using package main declaration, forming a single package of the executable).
Also check out the following related question: What is a sensible way to layout a Go project
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
Note: this question is related to this one, but two years is a very long time in Go history.
What is the standard way to organize a Go project during development ?
My project is a single package mypack, so I guess I put all the .go files in a mypack directory.
But then, I would like to test it during development so I need at least a file declaring the main package, so that I can do go run trypack.go
How should I organize this ? Do I need to do go install mypack each time I want to try it ?
I would recommend reviewing this page on How to Write Go Code
It documents both how to structure your project in a go build friendly way, and also how to write tests. Tests do not need to be a cmd using the main package. They can simply be TestX named functions as part of each package, and then go test will discover them.
The structure suggested in that link in your question is a bit outdated, now with the release of Go 1. You no longer would need to place a pkg directory under src. The only 3 spec-related directories are the 3 in the root of your GOPATH: bin, pkg, src . Underneath src, you can simply place your project mypack, and underneath that is all of your .go files including the mypack_test.go
go build will then build into the root level pkg and bin.
So your GOPATH might look like this:
~/projects/
bin/
pkg/
src/
mypack/
foo.go
bar.go
mypack_test.go
export GOPATH=$HOME/projects
$ go build mypack
$ go test mypack
Update: as of >= Go 1.11, the Module system is now a standard part of the tooling and the GOPATH concept is close to becoming obsolete.
jdi has the right information concerning the use of GOPATH. I would add that if you intend to have a binary as well you might want to add one additional level to the directories.
~/projects/src/
myproj/
mypack/
lib.go
lib_test.go
...
myapp/
main.go
running go build myproj/mypack will build the mypack package along with it's dependencies
running go build myproj/myapp will build the myapp binary along with it's dependencies which probably includes the mypack library.
I have studied a number of Go projects and there is a fair bit of variation. You can kind of tell who is coming from C and who is coming from Java, as the former dump just about everything in the projects root directory in a main package, and the latter tend to put everything in a src directory. Neither is optimal however. Each have consequences because they affect import paths and how others can reuse them.
To get the best results I have worked out the following approach.
myproj/
main/
mypack.go
mypack.go
Where mypack.go is package mypack and main/mypack.go is (obviously) package main.
If you need additional support files you have two choices. Either keep them all in the root directory, or put private support files in a lib subdirectory. E.g.
myproj/
main/
mypack.go
myextras/
someextra.go
mypack.go
mysupport.go
Or
myproj.org/
lib/
mysupport.go
myextras/
someextra.go
main/
mypack.go
mypage.go
Only put the files in a lib directory if they are not intended to be imported by another project. In other words, if they are private support files. That's the idea behind having lib --to separate public from private interfaces.
Doing things this way will give you a nice import path, myproj.org/mypack to reuse the code in other projects. If you use lib then internal support files will have an import path that is indicative of that, myproj.org/lib/mysupport.
When building the project, use main/mypack, e.g. go build main/mypack. If you have more than one executable you can also separate those under main without having to create separate projects. e.g. main/myfoo/myfoo.go and main/mybar/mybar.go.
I find very useful to understand how to organize code in Golang this chapter http://www.golang-book.com/11 of the book written by Caleb Doxsey
There doesn't seem to be a standard way of organizing Go projects but https://golang.org/doc/code.html specifies a best practice for most projects. jdi's answer is good but if you use github or bitbucket and you have additional libraries as well, you should create the following structure:
~/projects/
bin/
pkg/
src/
github.com/
username/
mypack/
foo.go
bar.go
mypack_test.go
mylib/
utillib.go
utillib_test.go
By doing it this way, you can have a separate repository for mylib that can be used for other projects and can be retrieved by "go get". Your mypack project can import your library using "github.com/username/mylib". For more information:
http://www.alexvictorchan.com/2014/11/06/go-project-structure/
Keep the files in the same directory and use package main in all files.
myproj/
your-program/
main.go
lib.go
Then run:
~/myproj/your-program$ go build && ./your-program
Let's explorer how the go get repository_remote_url command manages the project structure under $GOPATH. If we do a go get github.com/gohugoio/hugo It will clone the repository under
$GOPATH/src/repository_remote/user_name/project_name
$GOPATH/src/github.com/gohugoio/hugo
This is a nice way to create your initial project path. Now let's explorer what are the project types out there and how their inner structures are organized. All golang projects in the community can be categorized under
Libraries (no executable binaries)
Single Project (contains only 1 executable binary)
Tooling Projects (contains multiple executable binaries)
Generally golang project files can be packaged under any design principles such as DDD, POD
Most of the available go projects follows this Package Oriented Design
Package Oriented Design encourage the developer to keeps the implementation only inside it's own packages, other than the /internal package those packages can't can communicate with each other
Libraries
Projects such as database drivers, qt can put under this category.
Some libraries such as color, now follows a flat structure without any other packages.
Most of these library projects manages a package called internal.
/internal package is mainly used to hide the implementation from other projects.
Don't have any executable binaries, so no files that contains the main func.
~/$GOPATH/
bin/
pkg/
src/
repository_remote/
user_name/
project_name/
internal/
other_pkg/
Single Project
Projects such as hugo, etcd has a single main func in root level and.
Target is to generate one single binary
Tooling Projects
Projects such as kubernetes, go-ethereum has multiple main func organized under a package called cmd
cmd/ package manages the number of binaries (tools) that we want to build
~/$GOPATH/
bin/
pkg/
src/
repository_remote/
user_name/
project_name/
cmd/
binary_one/
main.go
binary_two/
main.go
binary_three/
main.go
other_pkg/