Related
I'm trying to debug Go implementation of ethereum(link), because my core interest is in developing new consensus algorithm (i.e. modify the open source Go code from github).
However, I'm having problem with the location/path of the source code. When I put the folder(i.e. go-ethereum) outside of the $GOPATH and then try to compile&debug geth(go-ethereum/cmd/geth/main.go) it shows the following error: Use of internal package is not allowed.
From that error message, I figured out that the import github.com/ethereum/go-ethereum was not importing my source, and instead it was getting code from internet(like other libraries). Which of course is definitely what I shouldn't do when I'm trying to modify the github.com/ethereum/go-ethereum package code.
So, my workaround was to clone the source code into $GOPATH/src/github.com/ethereum/go-ethereum and followed this answer, and Goland IDE started compiling&debugging without error (wasn't able to go build ./cmd/geth/main.go though due to error undefined: configFileFlag...)
Thereby now I've got a working debugger that can debug with my source code modification, but this doesn't look like ideal source code structure.
The question is:
Is putting source code inside $GOPATH(because of internals) a proper approach? If so, what if I was using go-ethereum package from another project?(Fortunately I'm not, but I'm curious) Do I have to stash&revert changes I made to the code?
Yes, the folder structure you ended up with is the right one.
Code should be under $GOPATH/src as you describe.
But note that $GOPATH is not a fixed folder in your system, you can have multiple projects under different folders and change the value of $GOPATH accordingly depending what you are working on (or have multiple console terminals open, each with its own $GOPATH value).
We have two versions of a go module hosted by different people. Let's call them example.org/devproject and company.com/project. Different programs obviously import the two projects by different import paths, as appropriate, depending on whether they want the version maintained by example.org or company.com. Because the modules have numerous internal packages, the source files themselves contain references to their own module's name in many import statements.
My question is how example.org can make it as easy as possible for company.com to pull from our git repository, because right now a straight git pull messes up the import statements. Note that stack overflow has similar questions on this prior to go modules. However, this question is specifically about projects with go.mod files using go 1.11 or later, so please do not refer me to non-module answers or suggest not using go modules yet, because that is a different question.
I've tried the following that unfortunately do not work super well:
Use some abstract module name like module self in the go.mod file and add replace self => example.org/devproject. Then internal packages can just import "self/subpackage", and only the one replace line needs to change between repositories. However, that means all including projects need to use a replace directive, and we'd need to convince company.com to change all of their imports relative to "self", which isn't super likely. More importantly, this totally breaks go get and is probably a bad idea.
Use semi-automated scripts to maintain a git branch pull-me that is always exactly one commit ahead of master but has all the import paths and go.mod file fixed up for company.com. This is what we currently do, but it's annoying because the scripts don't know how to parse go source code, so just blindly replace example.org/devproject with company.com/project everywhere, meaning there is no guarantee that just because master works pull-me will, too. We somehow can't get gomvpkg working in a go module outside of a gopath (it will change package statements, but not the imports, which are the tricky part, and won't recurse into all the subpackages).
Create a fake gopath, copy the master branch into it, use gomvpkg to move stuff, then copy the files back with a revised import path, then delete the fake gopath. I can't 100% rule this out, but I gave up after gomvpkg just kept refusing to do the right thing and I got frustrated that something so conceptually simple was so difficult to make work in practice.
Are there any other solutions we should be considering? Are there any tools like govmpkg that understand go modules and will rewrite import paths without also trying to move stuff around in a gopath?
As is probably clear, the dynamics here are that example.org is much more motivated to get stuff upstreamed than company.com is to pull from us, so ideally most of the hoop jumping would happen on the example.org side, and there would maybe be a one-time simple non-intrusive request to company.com to do something to make this work.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have a go project that is starting to become more complex, and want to lay the filesystem out in such a way to reduce pain.
Are there some good examples out there of what makes sense?
Update May 2013: the official documentation is in the section "Code organization"
Go code must be kept inside a workspace.
A workspace is a directory hierarchy with three directories at its root:
src contains Go source files organized into packages (one package per directory),
pkg contains package objects, and
bin contains executable commands.
The go tool builds source packages and installs the resulting binaries to the pkg and bin directories.
The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages.
bin/
streak # command executable
todo # command executable
pkg/
linux_amd64/
code.google.com/p/goauth2/
oauth.a # package object
github.com/nf/todo/
task.a # package object
src/
code.google.com/p/goauth2/
.hg/ # mercurial repository metadata
oauth/
oauth.go # package source
oauth_test.go # test source
Update July 2014: see "Structuring Applications in Go" from Ben Johnson
That article include tips like:
Separate your binary from your application
combining the main.go file and my application logic in the same package has two consequences:
It makes my application unusable as a library.
I can only have one application binary.
The best way I’ve found to fix this is to simply use a “cmd” directory in my project where each of its subdirectories is an application binary.
camlistore/
cmd/
camget/
main.go
cammount/
main.go
camput/
main.go
camtool/
main.go
Library driven development
Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library.
Sometimes you might want users to interact in multiple ways so you create multiple binaries.
For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version.
You can easily do this by organizing your project like this:
adder/
adder.go
cmd/
adder/
main.go
adder-server/
main.go
Users can install your “adder” application binaries with “go get” using an ellipsis:
$ go get github.com/benbjohnson/adder/...
And voila, your user has “adder” and “adder-server” installed!
Don’t go crazy with subpackages
Usually my project’s types are all very related so it fits better from a usability and API standpoint.
These types can also take advantage of calling unexported between them which keeps the API small and clear.
Group related types and code together in each file. If your types and functions are well organized then I find that files tend to be between 200 and 500 SLOC. This might sound like a lot but I find it easy to navigate. 1000 SLOC is usually my upper limit for a single file.
Organize the most important type at the top of the file and add types in decreasing importance towards the bottom of the file.
Once your application starts getting above 10,000 SLOC you should seriously evaluate whether it can be broken into smaller projects.
Note: that last practice isn't always good:
Sorry I just cant agree with this practice.
Separating type to files helps code management, readability, maintenancability, testability.
It may also ensure single responsibility and the follow of open/closed principle…
The rule for not allowing circular dependency is to force we have a clear structure of the packages.
(Alternative February 2013, regarding src only)
You can find the classic layout illustrated in "GitHub Code Layout":
The app and both libraries live on Github, each in its own repository.
$GOPATH is the root of the project - each of your Github repos will be checked out several folders below $GOPATH.
Your code layout would look like this:
$GOPATH/
src/
github.com/
jmcvetta/
useless/
.git/
useless.go
useless_test.go
README.md
uselessd/
.git/
uselessd.go
uselessd_test.go
README.md
Each folder under src/github.com/jmcvetta/ is the root of a separate git checkout.
That attracted some criticisms though, in this reddit page:
I highly recommend not structuring the repo the way you have, it'll break "go get", which is one of the most useful things about Go.
It's far better to write your code for people who do know Go, since they're most likely to be the ones compiling it.
And for people who don't, they'll at least get a feel for the language.
Put the main package in the root of the repo.
Have the assets in a subdirectory (to keep things neat).
Keep the meat of the code in a subpackage (in case anyone wants to reuse it outside your binary).
Include a setup script in the root of the repo so it's easy to find.
It's still only a two step process to download, build, install, and setup.:
"go get <your repo path>": downloads and installs the go code, with a subdir for the assets
$GOPATH/<your repo path>/setup.sh: distributes the assets to the right place and installs the service
I assume that with 'project' you don't mean a Go package but a software you develop.
Otherwise you can get help here and here.
However it's not so much different to writing packages for Go: Use packages, create a folder for each package and combine those packages in your application.
To build yourself an opinion, you can look at trending Go repositories on github: https://github.com/trending/go. Notable examples are cayley
and zeus.
The most popular scheme is probably to have a main Go file and many modules and submodules in their own directories. In case you have many meta files (doc, license, templates, ...) you may want to put the source code into a subdirectory. That's what I did so far.
There is a recommended approach from the authors of Golang that defines how to layout your code to work best with the go tools and to support source control systems
You should probably also have a look into this repo. It shows a lot of ideas how to structure go applications: https://github.com/golang-standards/project-layout
There are plenty of questions around this, including why you shouldn't use import "./my/path" and why it only works because some legacy go code requires it.
If this is correct, how do you handle encapsulation of a project and by extension github forks? In every other lang, I can do a github fork of a project, or git clone, and everything is encapsulated there. How do I get the same behaviour out of a go project?
Simple example using the go "hello world" example.
hello.go
package main
import ("fmt"
"github.com/golang/examples/stringutil")
func main() {
fmt.Printf(stringutil.Reverse("hello, world")+"\n")
}
The above works great. But if I want to use my own stringutil which is in a subdirectory and will compile to a single binary, I still need the complete path:
package main
import ("fmt"
"github.com/myrepo/examples/util/stringutil")
func main() {
fmt.Printf(stringutil.Reverse("hello, world")+"\n")
}
Now, if someone copies or forks my repo, it has a direct dependency on "github.com/myrepo/", even though this is used entirely internally!
What if there are 20 different files that import utils/? I need to change each one each time someone forks? That is a lot of extraneous changes and a nonsensical git commit.
What am I missing here? Why are relative paths such a bad thing? How do I fork a project that refers to its own subsidiary directories (and their packages) without changing dozens of files?
As for the reasoning behind not allowing relative imports, you can read this discussion for some perspective: https://groups.google.com/forum/#!msg/golang-nuts/n9d8RzVnadk/07f9RDlwLsYJ
Personally I'd rather have them enabled, at least for internal imports, exactly for the reason you are describing.
Now, how to deal with the situation?
If your fork is just a small fix from another project that will probably be accepted as a PR soon - just manually edit the git remotes for it to refer to your own git repo and not the original one. If you're using a vendoring solution like godep, it will work smoothly since saving it will just vendor your forked code, and go get is never used directly.
If your fork is a big change and you intend to remain forked, rewrite all the import paths. You can automate it with sed or you can use gofmt -r that supports rewriting of the code being formatted.
[EDIT] I also found this tool which is designed to help with that situation: https://github.com/rogpeppe/govers
I've done both 1 and 2 - when I just had a small bugfix to some library I just changed the remote and verndored it. When I actually forked a library without intent of merging my changes back, I changed all the import paths and continued to use my repo only.
I can also think of an addition to vendoring tools allowing automation of this stuff, but I don't think any of them support it currently.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have a go project that is starting to become more complex, and want to lay the filesystem out in such a way to reduce pain.
Are there some good examples out there of what makes sense?
Update May 2013: the official documentation is in the section "Code organization"
Go code must be kept inside a workspace.
A workspace is a directory hierarchy with three directories at its root:
src contains Go source files organized into packages (one package per directory),
pkg contains package objects, and
bin contains executable commands.
The go tool builds source packages and installs the resulting binaries to the pkg and bin directories.
The src subdirectory typically contains multiple version control repositories (such as for Git or Mercurial) that track the development of one or more source packages.
bin/
streak # command executable
todo # command executable
pkg/
linux_amd64/
code.google.com/p/goauth2/
oauth.a # package object
github.com/nf/todo/
task.a # package object
src/
code.google.com/p/goauth2/
.hg/ # mercurial repository metadata
oauth/
oauth.go # package source
oauth_test.go # test source
Update July 2014: see "Structuring Applications in Go" from Ben Johnson
That article include tips like:
Separate your binary from your application
combining the main.go file and my application logic in the same package has two consequences:
It makes my application unusable as a library.
I can only have one application binary.
The best way I’ve found to fix this is to simply use a “cmd” directory in my project where each of its subdirectories is an application binary.
camlistore/
cmd/
camget/
main.go
cammount/
main.go
camput/
main.go
camtool/
main.go
Library driven development
Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library.
Sometimes you might want users to interact in multiple ways so you create multiple binaries.
For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version.
You can easily do this by organizing your project like this:
adder/
adder.go
cmd/
adder/
main.go
adder-server/
main.go
Users can install your “adder” application binaries with “go get” using an ellipsis:
$ go get github.com/benbjohnson/adder/...
And voila, your user has “adder” and “adder-server” installed!
Don’t go crazy with subpackages
Usually my project’s types are all very related so it fits better from a usability and API standpoint.
These types can also take advantage of calling unexported between them which keeps the API small and clear.
Group related types and code together in each file. If your types and functions are well organized then I find that files tend to be between 200 and 500 SLOC. This might sound like a lot but I find it easy to navigate. 1000 SLOC is usually my upper limit for a single file.
Organize the most important type at the top of the file and add types in decreasing importance towards the bottom of the file.
Once your application starts getting above 10,000 SLOC you should seriously evaluate whether it can be broken into smaller projects.
Note: that last practice isn't always good:
Sorry I just cant agree with this practice.
Separating type to files helps code management, readability, maintenancability, testability.
It may also ensure single responsibility and the follow of open/closed principle…
The rule for not allowing circular dependency is to force we have a clear structure of the packages.
(Alternative February 2013, regarding src only)
You can find the classic layout illustrated in "GitHub Code Layout":
The app and both libraries live on Github, each in its own repository.
$GOPATH is the root of the project - each of your Github repos will be checked out several folders below $GOPATH.
Your code layout would look like this:
$GOPATH/
src/
github.com/
jmcvetta/
useless/
.git/
useless.go
useless_test.go
README.md
uselessd/
.git/
uselessd.go
uselessd_test.go
README.md
Each folder under src/github.com/jmcvetta/ is the root of a separate git checkout.
That attracted some criticisms though, in this reddit page:
I highly recommend not structuring the repo the way you have, it'll break "go get", which is one of the most useful things about Go.
It's far better to write your code for people who do know Go, since they're most likely to be the ones compiling it.
And for people who don't, they'll at least get a feel for the language.
Put the main package in the root of the repo.
Have the assets in a subdirectory (to keep things neat).
Keep the meat of the code in a subpackage (in case anyone wants to reuse it outside your binary).
Include a setup script in the root of the repo so it's easy to find.
It's still only a two step process to download, build, install, and setup.:
"go get <your repo path>": downloads and installs the go code, with a subdir for the assets
$GOPATH/<your repo path>/setup.sh: distributes the assets to the right place and installs the service
I assume that with 'project' you don't mean a Go package but a software you develop.
Otherwise you can get help here and here.
However it's not so much different to writing packages for Go: Use packages, create a folder for each package and combine those packages in your application.
To build yourself an opinion, you can look at trending Go repositories on github: https://github.com/trending/go. Notable examples are cayley
and zeus.
The most popular scheme is probably to have a main Go file and many modules and submodules in their own directories. In case you have many meta files (doc, license, templates, ...) you may want to put the source code into a subdirectory. That's what I did so far.
There is a recommended approach from the authors of Golang that defines how to layout your code to work best with the go tools and to support source control systems
You should probably also have a look into this repo. It shows a lot of ideas how to structure go applications: https://github.com/golang-standards/project-layout