First of all and to be clear I come from the Java world, and I have been programming on go for a while and I really love it.
I have a small question about the packaging system and the imports, if I am importing a library which uses another library and I am already using that library in my project, how can I eliminate the duplication(if its possible),
in other words:
A is a the main program, C and B are libraries, and then:
C was added to A
B uses C
and then
B was also added to A
AProject/
src/
LibC/
src/
somefiles.go
LibB/
src/
LibC
somefiles.go
app.go
So now I have two libraries of C one in A since the beginning and one in B because B is dependent on C.
I know its a little bit confusing but in the Java world we have Ant and Maven and those build tools make it really easy to us to handle the dependencies.
Any thoughts?
In Go, there is no duplication of packages.
First, you should read about Go workspaces in How to Write Go Code.
From your question, your directory structure should look something like this:
gopath (gopath is the path of a directory in your $GOPATH list)
├── bin
│ └── projecta
├── pkg
│ └── linux_amd64
│ └── projecta
│ ├── libb.a
│ └── libc.a
└── src
└── projecta
├── a.go
├── libb
│ └── b.go
└── libc
└── c.go
Where,
gopath/src/projecta/a.go:
package main
import (
"projecta/libb"
"projecta/libc"
)
func a() {
libb.B()
libc.C()
}
func main() { a() }
gopath/src/projecta/libb/b.go:
package libb
import (
"projecta/libc"
)
func B() { libc.C() }
gopath/src/projecta/libc/c.go:
package libc
func C() {}
If you are talking about third-party libraries, in go is very simple to do that,
just put the import in your source code, like:
import "github.com/somepackage/somelib"
and from the command line in your working directory run:
go get
the the source code of the libraries will be downloaded in the src directory of your $GOPATH.
If you want to create your own lib instead, just create the folder named as the lib in $GOPATH/src and put the code in this folder.
The folders structure is:
$GOPATH/
src/
github.com/
somepackage/
somelib/
somelib.go
yourlibB/
yourlibB.go -> //import somelib here
yourlibC/
yourlibC.go -> //import libB here
yourmainprogramA/
yourmainprogramA.go -> //import somelib, libC and libB here
Related
I have 2 grpc services (service1 and service2) that interact with each other and on some cases the rpc response of service1 will consists of a struct defined in service2, after going to several situations where duplication is inevitable, i figure that as the services grow these will be hard to manage, so i restructure the proto files into something like this for now
.
├── app
...
├── proto
│ ├── service1
│ │ ├── service1.access.proto
│ │ ├── service1.proto
│ ├── service2
│ │ ├── service2.access.proto
│ │ └── service2.proto
│ └── model
│ ├── model.service1.proto
│ └── model.service2.proto
└── proto-gen // the protoc generated files
├── service1
│ ├── service1.access.pb.go
│ └── service1.pb.go
├── service2
│ ├── service2.access.pb.go
│ └── service2.pb.go
└── model
├── model.service1.pb.go
└── model.service2.pb.go
service1 needs to import the model definition on model/model.service2.proto, so i am importing it like this
import "model/model.service2.proto";
option go_package = "proto-gen/service1";
and i generate the .pb.go files, using this protoc command
ls proto | awk '{print "protoc --proto_path=proto proto/"$1"/*.proto --go_out=plugins=grpc:."}' | sh
the command generates the .pb.go files just fine, but the code on service1.access.pb.go doesn't seem to import the model correctly, and i don't know if it related or not but when i run the app, it throws this error
cannot load model: malformed module path "model": missing dot in first path element
i spent a few hours now googling on how can i properly import another proto file, i can't seem to find any solution
The reason you got that error about model is because the generated files use the go_package of the imported file, and model is not a valid import path. You have to convince the generated file to use the full import path of the package.
This is how I did it for my source tree: I have a similar tree of proto files importing each other. If your module is named, say github.com/myapp, then run protoc with --proto-path=<directory containing github.com>, import other proto files using full path, that is github.com/myapp/proto/service1/service1.proto, and in service1.proto, define go_package = service1. This setup writes the import paths correctly in my case.
Before settling into this solution, I was using go_package=<full path to proto>, so you might give that a try as well.
Building on Burak Serdar, I want to provide my implementation.
Set the package on the proto you want to import similar to this where the location is your full path. My path is generally github.com/AllenKaplan/[project]/[package]/proto/
option go_package = [path];
In the file you wish to import to add an import. My path is generally [package]/proto/[package].proto
import = [path from protoc proto path]
The last part is the protoc command where you must define the protopath in a way that connects the import path and option go_package path
if executing from the github.com/AllenKaplan/[project] directory, I would call
protoc -I. --go_out=./[package]/proto [package]/proto/[package].proto
-I. === --proto_path.
the -I. sets the protopath to the entire project
One note, when calling protoc on your .proto files that you are importing, you will want to add source_relative: to the output will ensure the output is from the root with a set package.
My implementation of the imported protoc when called from github.com/AllenKaplan/[project]/[package]
protoc -I./proto --go_out=paths=source_relative:./proto [package].proto
I was also facing a similar issue while importing. Had changed the .protoc file option package with the following.
option go_package = "./;proto-gen/service1";
The first param means relative path where the code you want to generate.
The path relative to the --go_out , you set in your command.
I have a repository structure as follows :-
xyz/src
1. abc
- p
- q
- r
2. def
- t
- u
- v
3. etc
- o
- m
- n
I have created a .mod file in src and run go build ./...
Except for local packages everything is fine. So if abc/p is being used in def then it throws the following exception :- cannot find module providing package abc/p. The idea behind keeping the .mod file in src package was to make sure the path is being found from where the mod file is located. Can anyone suggest where should the mod file ideally should be? also i tried placing it one directory above in xyz but still same issue as well as i created one for each sub directory. I am bit confused on this. Will I have to create separate repository for abc and etc. But considering gopath which earlier used to work for the same I think module should also be able to do the same. Any suggestions?
The most common and easiest approach is a single go.mod file in your repository, where that single go.mod file is placed in the root of your repository.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
The Modules wiki says:
For example, if you are creating a module for a repository
github.com/my/repo that will contain two packages with import paths
github.com/my/repo/foo and github.com/my/repo/bar, then the first
line in your go.mod file typically would declare your module path as
module github.com/my/repo, and the corresponding on-disk structure
could be:
repo/
├── go.mod <<<<< Note go.mod is located in repo root
├── bar
│ └── bar.go
└── foo
└── foo.go
In Go source code, packages are imported using the full path including
the module path. For example, if a module declared its identity in its
go.mod as module github.com/my/repo, a consumer could do:
import "example.com/my/repo/bar"
That imports package bar from the module github.com/my/repo.
I have a single go.mod in the root of my go application. I am using the following structure inspired by Kat Zien - How Do You Structure Your Go Apps
At the minute one of my applications looks like this
.
├── bin
├── cmd
│ ├── cli
│ └── server
│ └── main.go
├── pkg
│ ├── http
│ │ └── rest
| │ # app-specific directories excluded
│ └── storage
│ └── sqlite
All packages are imported via their full path, i.e. import "github.com/myusername/myapp/pkg/http/rest" otherwise it causes problems all over the place and this was the one change I had to make going from $GOPATH to go mod.
go mod then handles all the dependencies it discovers properly as far as I've discovered so far.
I'm new to go modules, and am taking them for a spin in a new project which I'm trying to model after the structure described here
Here is an example of my directory structure:
.
├── cmd
│ └── app_name
│ └── main.go
├── go.mod
├── go.sum
├── internal
│ └── bot
│ └── bot.go
└── pkg
├── website_name
│ ├── client.go
│ ├── client.options.go
│ ├── server.go
│ └── server.options.go
└── lib
└── lib.go
Is this idiomatically correct? I know there's not a whole lot of consensus out there, but I'd like to follow best practices.
When I run go build I get 'unexpected module path "github.com/ragurney/app_name/cmd/app_name"', but when I run go build ./... it works. Why?
When I move main.go to the top level everything works as expected. Should I just not use the /cmd pattern with modules?
To answer your first question, its completely opinionated and whatever you like best that is also easy to understand for others you should go with (I think it's fine).
To answer your second question the reason go build ./... works as opposed to go build from the root directory is because ./... starts in the current directory (the root) and searches for all program entry-points and builds them. When you move main.go to the root directory, with this new information, go build working then makes sense, as its only looking in the current directory.
You can explicitly say go build ./cmd/app_name which would also work.
Your application structure works perfectly fine with modules, as I use something very similar to it (https://www.ardanlabs.com/blog/2017/02/package-oriented-design.html) and modules work very well for me.
from what i can tell there is nothing wrong with your project structure. What has worked for me is to run the go build/run command from the project root
eg.
go run github.com/username/project/cmd/somecommand
go build -o somebinary github.com/username/project/cmd/somecommand
I prefer to add the specific file to build, there are some projects with more than one executable
go build -o app ./cmd/server/main.go
I’ve read as many docs and StackOverflow articles as I can find, yet I am having no luck importing using the new vendor feature in Go 1.6.
Here's a sample project I put together with Goji to test. The directory structure is as such:
.
└── src
├── main.go
└── vendor
└── github.com
└── zenazn
└── goji
├── LICENSE
├── README.md
├── bind
├── default.go
├── example
├── goji.go
├── graceful
├── serve.go
├── serve_appengine.go
└── web
And main.go, the sole file in the project, is as such:
package main
import (
"fmt"
"net/http"
"github.com/zenazn/goji"
"github.com/zenazn/goji/web"
)
func hello(c web.C, w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, %s!", c.URLParams["name"])
}
func main() {
goji.Get("/hello/:name", hello)
goji.Serve()
}
My environment variables are as such:
export GOPATH=~/.go
export GOBIN=$GOPATH/bin
export PATH=$PATH:/usr/local/opt/go/libexec/bin:$GOBIN
I’ve tried the most simple build commands, with no luck:
go run ./src/main.go
go build ./src/main.go
I’ve also attempted to build with:
$GOPATH=`pwd`
...to no avail. Am I totally missing something? Any advice is appreciated.
I suggest you to read https://golang.org/doc/code.html. It requires a day or two to digest but after you understand how go tools work with the source code and GOPATH it is really easy to use them.
Back to your question. To build a simple go program you need to:
create directory under $GOPATH/src, e.g. mkdir $GOPATH/src/myprogram
put all the source code (including vendor directory) there: $GOPATH/src/myprogram/main.go, $GOPATH/src/myprogram/vendor.
run go install myprogram to build your application and put the resulting myprogram binary to $GOPATH/bin/myprogram
I try understand how to organize go code using "internal" packages. Let me show what the structure I have:
project/
internal/
foo/
foo.go # package foo
bar/
bar.go # package bar
main.go
# here is the code from main.go
package main
import (
"project/internal/foo"
"project/internal/bar"
)
project/ is outside from GOPATH tree. Whatever path I try to import from main.go nothing works, the only case working fine is import "./internal/foo|bar". I think I do something wrong or get "internal" package idea wrong in general. Could anybody make things clearer, please?
UPDATE
The example above is correct the only what I need was to place project/ folder under $GOPATH/src. So the thing is import path like the project/internal/foo|bar is workable if we only import it from project/ subtree and not from the outside.
With modules introduction in Go v1.11 and above you don't have to specify your project path in $GOPATH/src
You need to tell Go about where each module located by creating go.mod file. Please refer to go help mod documentation.
Here is an example of how to do it:
project
| go.mod
| main.go
|
\---internal
+---bar
| bar.go
| go.mod
|
\---foo
foo.go
go.mod
project/internal/bar/go.mod
module bar
go 1.14
project/internal/bar/bar.go
package bar
import "fmt"
//Bar prints "Hello from Bar"
func Bar() {
fmt.Println("Hello from Bar")
}
project/internal/foo/go.mod
module foo
go 1.14
project/internal/foo/foo.go
package foo
import "fmt"
//Foo prints "Hello from Foo"
func Foo() {
fmt.Println("Hello from Foo")
}
project/main.go
package main
import (
"internal/bar"
"internal/foo"
)
func main() {
bar.Bar()
foo.Foo()
}
Now the most important module
project/go.mod
module project
go 1.14
require internal/bar v1.0.0
replace internal/bar => ./internal/bar
require internal/foo v1.0.0
replace internal/foo => ./internal/foo
Couple things here:
You can have any name in require. You can have project/internal/bar if you wish. What Go think it is URL address for the package, so it will try to pull it from web and give you error
go: internal/bar#v1.0.0: malformed module path "internal/bar": missing dot in first path element
That is the reason why you need to have replace where you tell Go where to find it, and that is the key!
replace internal/bar => ./internal/bar
The version doesn't matter in this case. You can have v0.0.0 and it will work.
Now, when you execute your code you will have
Hello from Bar
Hello from Foo
Here is GitHub link for this code example
The packages have to be located in your $GOPATH in order to be imported. The example you gave with import "./internal/foo|bar" works because it does a local-import. internal only makes it so code that doesn't share a common root directory to your internal directory can't import the packages within internal.
If you put all this in your gopath then tried to import from a different location like OuterFolder/project2/main.go where OuterFolder contains both project and project2 then import "../../project/internal/foo" would fail. It would also fail as import "foo" or any other way your tried due to not satisfying this condition;
An import of a path containing the element “internal” is disallowed if
the importing code is outside the tree rooted at the parent of the
“internal” directory.
Now if you had the path $GOPATH/src/project then you could do import "foo" and import "bar" from within $GOPATH/src/project/main.go and the import would succeed. Things that are not contained underneath project however would not be able to import foo or bar.
below way is more scalable, especially when you plan to build multiple binaries
github.com/servi-io/api
├── cmd/
│ ├── servi/
│ │ ├── cmdupdate/
│ │ ├── cmdquery/
│ │ └── main.go
│ └── servid/
│ ├── routes/
│ │ └── handlers/
│ ├── tests/
│ └── main.go
├── internal/
│ ├── attachments/
│ ├── locations/
│ ├── orders/
│ │ ├── customers/
│ │ ├── items/
│ │ ├── tags/
│ │ └── orders.go
│ ├── registrations/
│ └── platform/
│ ├── crypto/
│ ├── mongo/
│ └── json/
The folders inside cmd/ represents the number of binaries you want to build.
for more
Also to check: When you are using your externally imported object types: make sure you are prefixing them with the namespace they're in. As a golang newbie, I didn't know I had to do that, and was wondering why is VS Code just removing my import (for not being used) when I saved. It's because I had to prefix the imported object with the namespace name:
Example:
import (
"myInternalPackageName" // works fine as long as you follow all instructions in this thread
)
//Usage in code:
myInternalPackageName.TheStructName // prefix it, or it won't work.
If you don't put the namespace prefix before the object/struct name, VS code just removes your import for being unused, and then you still have the error: "Can't find TheStructName"... That was very confusing, and I had to do a build without VS code through the command line to understand that.
The point is: I had to specify the package prefix when actually using the struct from that internal package.
If you don't want to use the qualifier prefix when using imported objects, use:
import . "thePath" // Can use contents without prefixing.
Reference: What does the '.' (dot or period) in a Go import statement do?