How to setup protobuf, grpc and Go modules in a monorepo? - go

Question

I'm trying to create a microservice-based API in Go, very basic, just to learn some stuff. I have three main goals I have to achieve.
Monorepo.
Be able to use different back-end languages as need arises (ie. one microservice in Go, two in Node.js, one in Python).
Use gRPC.
Currently my structure looks like this (and is stored in ~/Projects/tkg, outside of GOPATH).
Each "service" should be a self-contained application written in a "whatever". As you can see I have a Go service and a React front-end application. Additionally there is a Makefile there that I want to use for building stuff, but I might move to shell scripts, Docker, whatever. Doesn't matter.
So now the question. How can I make generated proto files play well with this setup? I think I don't understand something about Go modules and packages because I cannot set it so articles.go (from cmd) can access the generated api/article.pb.go. How to do it?
// services/articles/go.mod
module tkg/services/articles
go 1.12
require (
github.com/golang/protobuf v1.3.2
google.golang.org/grpc v1.22.1
)
// services/articles/cmd/article.go
package main
import (
pb "tkg/services/articles/api/article"
)
type repository interface {
Create(*pb.Article) (*pb.Article, error)
}
func main() {
}
// services/articles/api/article.proto
syntax = "proto3";
package article;
option go_package = "tkg/services/articles/api/article";
...
// Makefile
build:
protoc services/articles/api/article.proto --go_out=.
I have tried various different package names in go.mod, different go_packages in the proto file, I had tried different protoc commands and paths. I bet this is silly and it's very obvious to someone who is well-versed in Go, but for someone from Node.js backgroud like me the inability to do import "../api/article.pb.go" is infuriating. :(
The error I am getting is: could not import tkg/services/articles/api/article (no parsed files for package tkg/services/articles/api/article). Of course with different values for package names. I've been trying to solve it for two days now.
How would you approach this problem?

Answer 1

If you are generating the .pb.go file in the same directory as the .proto file (recommended), then your import path should read:
import (
pb "tkg/services/articles/api"
)
(not tkg/services/articles/api/article.)
Go packages are collections of .go files in a single directory - each with the same package XYZ first line - where XYZ is the package name. So when importing a package, one uses the package base directory - not including any .go filenames.
Edit: (this was too long to fit into a comment):
I would step back and think about the base-directory of your entire project.
Standard go packages are usually single words like time, sync etc. because they are part of go's standard library. All other packages should have a full internet path. These typically match a git repo address (e.g. "github.com/boltdb/bolt") - but not always e.g. ("gopkg.in/yaml.v2"). This is actually superior to the centrally-hosted NPM package model - as it easily allows for pulling packages from any repo host. Also since repos are cloned to local disk - those same paths can exist on your local disk first (before they've been hosted on the internet, say for development purposes).
So I would suggest naming your local base-directory something like:
github.com/myname/myproj/tkg/services/...
and then ensure you import packages based on this directory structure e.g.
import "github.com/myname/myproj/tkg/services/api"
If a go build does not pick up the generated (.pb.go) code, then there is something up with your GOPATH or if using the new go-modules your go.mod setup.