How to structure multi language repository to work reasonably with golang? - go

I want to make a library with few language bindings, one of them being golang. It would make sense to have it all in one repository, because for example test data can be shared between them. But I'm not sure how to structure it. I've tried is
+ $ tree .
.
└── go
├── go.mod
└── foo.go
1 directory, 2 files
+ $ cat go/go.mod
module github.com/bar/foo/go
go 1.13
+ $ cat go/foo.go
package foo
func Foo() int {
return 1;
}
Then I tried to use it
+:( $ cat test.go
package main
import (
"github.com/bar/foo/go"
)
func main() {
fmt.Println(foo.Foo())
}
But it failed with
+ $ go run ./test.go
go: finding github.com/bar/foo latest
build command-line-arguments: cannot load github.com/bar/foo/go: module github.com/bar/foo#latest (v0.0.0-20191019071519-05aedbdfa600) found, but does not contain package github.com/bar/foo/go
I'm fairly new to golang, so could someone please advice on how are projects like this usually structured?

First of all, in most cases it is a bad idea to name a package after the programming language. What does the package actually DO? Remember, documentation and package layout should be optimized for the user.
For multi-part projects, I personally tend to set up a new organization say foo and create a bar repo for the first part, say the server written in Java, which then would be accessible via https://github.com/foo/bar. Next, for the client written in Go, you set up the next repo baz, which then would be accessible via https://github.com/foo/baz - your go module.
Last, but not least, you can have a „distribution“ repository, which puts the organization‘s projects through a CI pipeline to ensure interoperability.
This approach has multiple advantages, imho: You make it easier for the user to explore the ecosystem, the repos become sufficiently independent while still being tied together by the organization and the individual repos become smaller.

Related

Go test results in go: cannot find main module, but found .git/config in /Users/dp/Documents

Im just getting started with Golang and am trying to run unit tests. When running go test, I end up getting the following output from the terminal
go: cannot find main module, but found .git/config in /Users/dp/Documents
to create a module there, run:
cd ../.. && go mod init
My file structure is fairly simple, as is as follows
├── CARDS
│ ├── .vscode
│ ├── main.go
│ ├── test.go
│ ├── deck_test.go
│ └── my_cards.txt
the contents of deck_test.go are
package main
import "testing"
func TestNewDeck(t *testing.T) {
d := newDeck()
if len(d) != 52 {
t.Errorf("Length is not 52, got %v", len(d))
}
if d[0] != "A of D" {
t.Errorf("First card is not A of D, got %v", d[0])
}
if d[len(d)-1] != "K of C" {
t.Errorf("Last card is not K of C, got %v", d[len(d)-1])
}
}
Any insights would be of help!
Although not required, it's generally advisable to init when you're inside of a Git (or other VCS) repository so that modules can lean on your remotes information to determine the name of the module correctly, for example:
git init
git remote add origin https://github.com/syntaqx/dacode
go mod init
Allows go mod to understand my modules name is likely intended to be github.com/syntaqx/dacode.
Alternatively, and I often do this for sake of not needing to do things in any particular order, you can specify the module name:
go mod init dacode # valid, but..
go mod init github.com/syntaqx/dacode # is generally better, because it describes my remote
By specifying it, modules can just initialize in whatever directory I'm in without having to magic anything from the code.
Note: While you're starting out especially, I highly recommend naming your modules the same naming structure you would use for a repository. By doing so, you're allowing commands like:
go get github.com/syntaqx/dacode
To function correctly without having to do any of the internals yourself. As you learn more about how they work, you can decide if you want to keep that convention or go against the grain, but it's likely more sane to be consistent.
Hope this helped!
I am not using go modules (I know, I should). I got this error after upgrading from older Go versions to 1.16. From release notes:
Note that Go 1.16 requires use of Go modules by default, now that, according to our 2020 Go Developer Survey, 96% of Go developers have made the switch. We recently added official documentation for developing and publishing modules.
To still allow working as before, change it to as it was before. Add this in your .bashrc:
export GO111MODULE=auto

Go 'mod init' creating new folders? what is the significance of path?

Just 3 days experience in Go language. Hope an example will be more apt to understand my confusion.
root#freebsd1:/usr/home/arun/go-start/src/test2 # go mod init f1/f2/f3/f4/f5/hello
go: creating new go.mod: module f1/f2/f3/f4/f5/hello
root#freebsd1:/usr/home/arun/go-start/src/test2 #
Here in above example go mod init is creating all these folders(f1/f2/f3/f4/f5/hello)?. I searched a lot, couldn't find any such folders in system. Then what is the significance of this path.
Even though below command will not run if this path is not mentioned as it is
# go install f1/f2/f3/f4/f5/hello
--:EDIT:--
May be this will help someone later ( Just walk through the steps to understand this in a proper way, especially for newbies )
I am planning to create a program 'calculator' and will upload in GitHub later.
I will keep the functions in different packages like sum,multiply etc
first step #go mod init github.com/go-arun/calculator( Don't confuse here , this is just an assumption that, in future I may create a repository in github for this project )
created folder sum(one of the package folder , and created sum.go inside )
See those in by system:
1.
root#debian1:/home/arun/lab# go mod init github.com/go-arun/calculator
go: creating new go.mod: module github.com/go-arun/calculator
root#debian1:/home/arun/lab# cat go.mod
module github.com/go-arun/calculator
go 1.15
2.
root#debian1:/home/arun/lab# cat sum/sum.go
package sum
import "fmt"
func Sum(num1,num2 int)(){
fmt.Println(num1+num2)
}
3.
root#debian1:/home/arun/lab# cat main.go
package main
import(
"github.com/go-arun/calculator/sum"
)
func main(){
n1 := 10
n2 := 10
sum.Sum(n1,n2)
}
4.
root#debian1:/home/arun/lab# tree
.
|-- go.mod
|-- main.go
`-- sum
`-- sum.go
go mod init does not create those folders. You pass the "module path" to go mod init which is recorded in the go.mod file it creates.
The "module path" is the import path prefix corresponding to the module root. The module path and the relative path to the module root together form the complete import path which must be unique in an app.
So for example if your module contains a folder named foo (and a package foo in it), it is imported by a path being modulepath/foo. In your case it would be f1/f2/f3/f4/f5/hello/foo.
It is allowed for moduleA to contain a foo package, and also for moduleB to have a foo package. When used / imported, first would be imported like moduleA/foo the latter like moduleB/foo, so it's unambiguous which one you're importing. The module path is like a namespace.
It's recommended to use a module path that corresponds to a repository you plan or will publish your module to, so when you do, go get will be able to automatically fetch, build and install your module. For example you may choose a module path github.com/bob/hello, so when you publish your module, everyone can get it by simply using import "github.com/bob/hello" in their app.
Also note that you don't need to publish your code to a remote repo before you can build it. But it's still recommended to follow this pattern so you'll have less work to make it work in the future if you decide to publish it. Nothing to lose here.
More in the docs: Command go: Defining a module
Also: How to Write Go Code: Code organization

How to setup protobuf, grpc and Go modules in a monorepo?

I'm trying to create a microservice-based API in Go, very basic, just to learn some stuff. I have three main goals I have to achieve.
Monorepo.
Be able to use different back-end languages as need arises (ie. one microservice in Go, two in Node.js, one in Python).
Use gRPC.
Currently my structure looks like this (and is stored in ~/Projects/tkg, outside of GOPATH).
Each "service" should be a self-contained application written in a "whatever". As you can see I have a Go service and a React front-end application. Additionally there is a Makefile there that I want to use for building stuff, but I might move to shell scripts, Docker, whatever. Doesn't matter.
So now the question. How can I make generated proto files play well with this setup? I think I don't understand something about Go modules and packages because I cannot set it so articles.go (from cmd) can access the generated api/article.pb.go. How to do it?
// services/articles/go.mod
module tkg/services/articles
go 1.12
require (
github.com/golang/protobuf v1.3.2
google.golang.org/grpc v1.22.1
)
// services/articles/cmd/article.go
package main
import (
pb "tkg/services/articles/api/article"
)
type repository interface {
Create(*pb.Article) (*pb.Article, error)
}
func main() {
}
// services/articles/api/article.proto
syntax = "proto3";
package article;
option go_package = "tkg/services/articles/api/article";
...
// Makefile
build:
protoc services/articles/api/article.proto --go_out=.
I have tried various different package names in go.mod, different go_packages in the proto file, I had tried different protoc commands and paths. I bet this is silly and it's very obvious to someone who is well-versed in Go, but for someone from Node.js backgroud like me the inability to do import "../api/article.pb.go" is infuriating. :(
The error I am getting is: could not import tkg/services/articles/api/article (no parsed files for package tkg/services/articles/api/article). Of course with different values for package names. I've been trying to solve it for two days now.
How would you approach this problem?
If you are generating the .pb.go file in the same directory as the .proto file (recommended), then your import path should read:
import (
pb "tkg/services/articles/api"
)
(not tkg/services/articles/api/article.)
Go packages are collections of .go files in a single directory - each with the same package XYZ first line - where XYZ is the package name. So when importing a package, one uses the package base directory - not including any .go filenames.
Edit: (this was too long to fit into a comment):
I would step back and think about the base-directory of your entire project.
Standard go packages are usually single words like time, sync etc. because they are part of go's standard library. All other packages should have a full internet path. These typically match a git repo address (e.g. "github.com/boltdb/bolt") - but not always e.g. ("gopkg.in/yaml.v2"). This is actually superior to the centrally-hosted NPM package model - as it easily allows for pulling packages from any repo host. Also since repos are cloned to local disk - those same paths can exist on your local disk first (before they've been hosted on the internet, say for development purposes).
So I would suggest naming your local base-directory something like:
github.com/myname/myproj/tkg/services/...
and then ensure you import packages based on this directory structure e.g.
import "github.com/myname/myproj/tkg/services/api"
If a go build does not pick up the generated (.pb.go) code, then there is something up with your GOPATH or if using the new go-modules your go.mod setup.

Are there conventions for Go module names in mono-repos with multiple modules?

In a multi-module repository, should a module name (set via the go.mod module directive) follow the conventions of package naming?
E.g. module github.com/org-name/repo-name/path/to/module-dir
I understand that, whatever the module is named, the packages within the module refer to each other using the module name as prefix. But, from outside the module, there seems to problems if the module name is set to something other than the <host><path-within-repo> pattern. get-ing a package included in the module then gives messages about unrecognized import path.
Is there any cause to name a module differently than <host><path-within-repo> ?
There isn't any hard requirement for referencing modules, although it always good practice to use the domain/repo pattern. So, if you want to reference other modules locally that are not in the GOPATH, you can use the replace directive.
https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive
replace also can be used to inform the go tooling of the relative or absolute on->disk location of modules in a multi-module project, such as:
replace example.com/project/foo => ../foo
Let's say we have the following structure:
├── .gitignore
├── pkg1
│   ├── go.mod
│   └── main.go
└── pkg2
├── go.mod
└── utils.go
pkg1/main.go
package main
import (
"fmt"
"local/pkg2"
)
func main() {
fmt.Println(pkg2.Add(1, 2))
}
pkg1/go.mod
module local/pkg1
go 1.12
require local/pkg2 v0.0.0
replace local/pkg2 => ../pkg2
pkg2/utils.go
package pkg2
func Add(a, b int) int {
return a + b
}
pkg2/go.mod
module local/pkg2
go 1.12
Running:
cd pkg1
go run main.go
You get:
3
If you want to be able to go get a module, it should follow the <host>/repo/path/within/repo pattern.
However, I would suggest stepping back and asking if you really do want a multi-module repo. It adds substantial complexity, it is hard to get right, and usually means more work on an on-going basis.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
For more details, see "Organize local code in packages using Go modules".

Importing local library and files in an application

I'm new to Go (but not at programming), I love the language but I have a bit of trouble fully understanding the way I'm supposed to make internal libraries in an application through packages. For reference, getting external packages and then importing/using them is fine.
Let's say I'm making an application A.
/home/me/A/a.go (package main)
Then, I realize a.go start to be rather big, so I cut it into two parts
/home/me/A/a.go (package main)
/home/me/A/b.go (package main)
How am I supposed to import/include b.go from a.go to make its function available ?
As a continuation of the question, in the A I'm manipulation lots of objects O, so I figure it would be a lot better if I just give them their own package and encapsulate the functionalities in a public/exported api. How do I do that ?
I've tried creating ./lib/o.go (package o) and import lib/o but I keep getting error like
./a.go:6: imported and not used: "o"
./a.go:43: undefined: o
I have no GOPATH in my env but I tried export GOPATH=$GOPATH:/home/me/A and it didn't change the result.
I've tried to read the article on "go layout" but it felt a bit too overwhelming at once and I would really love a simpler explanation of that one "small" step I am trying to make
Thanks !
GOPATH/src/me/a/a.go:
package main
func main() {
test()
}
GOPATH/src/me/a/test.go:
package main
import "fmt"
func test() {
fmt.Println("test func !")
}
Exec:
$ go run a.go
# command-line-arguments
./a.go:4: undefined: test
EDIT: got my answer here: https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/qysy2bM_o1I
Either list all files in go run (go run a.go test.go) or use go build and run the resulting executable.
You're trying to use the Go build system while not following the necessaary required directory layouts. You will benefit a lot from reading this document.
In short, these are, wrt the go tool, the show stoppers:
You must have a valid, exported GOPATH
Package files with import path "example/foo" must be located in the $GOPATH/src/example/foo directory.
For more details please see the above linked article.

Resources