I'm trying to set a repo for my proto files.
I have the following structure:
buf.gen.yaml
folderA
buf.yaml
v1
file1.proto
folderB
buf.yaml
v1
file2.proto
file1.proto needs to import a message from file2.proto, I can generate the protos code without any issues with the following:
buf generate
File A imports file B with the following
import "folderB/v1/file2.proto";
I tried adding a buf.work.yaml with the following:
version: v1
directories:
- folderA
- folderB
but I still get folderA/v1/file1.proto:7:8:folderB/v1/file2.proto: does not exist when I try to lint, but can still generate the code.
Can someone explain to me what am I missing?
Thanks.
Related
In my protobuf generated go code, the package is:
package __
What does the double underscore mean, does it means the same as folder name ?
Is there a document for this, I searched but didn't found any. And the code can compile without error.
Yes it means the same directory. Let's look at the following code.
Directory Structure
.
├── go.mod
├── greet
│ └── greet.go
└── main.go
Content in greet.go
package __
import "fmt"
func Hello(name string) {
fmt.Printf("Hello %s\n", name)
}
Content in main.go
package main
import greet "playground/greet"
func main() {
greet.Hello("Eric")
}
Current Directory
$ pwd
/Users/thedatageek/Codes/go-playground
Unfortunately I also couldn't find any docs for go.
But it seems that its kinda good thing. You really don't need to name the package. You just name the directory and the package name would be automatically the same.
Note: This is definitely not the grpc or protobuf thing. It is however a customary that if you have generated proto stub from a proto file and if you add some additional utility file you may put those into a dir and then import it directly via directory name. For example the following github repos
https://github.com/Ash110/gRPC-Logger
https://github.com/dist1ll/cache-prototype
https://github.com/kamensotirov99/int-gateway
https://github.com/rachaelyychen/go-gee
https://github.com/suvvm/ToadOCREngine
https://github.com/denyami/drawing-api
I encounter an issue with Go Module management and a generation of protobuffers (using go1.16, protoc-gen-go#latest).
I have this project structure:
subproj
├── go.mod (module company.tld/proj/subproj)
├── subproj.go (entry point : package main)
├── proto (folder containing .proto files)
├── packageFolder
| └── file1.go (package packageFolder)
└── Makefile (used to generate *.pb.go and build subproj binary)
The proto folder is used by other projects (obviously...) (via git submodule).
Protos are like the following:
syntax = "proto3"
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
package entity.proj
...
because of different version of messages, few protobuffer files need to be in another "namespace":
option go_package = "company.tld/proj/projpb/other";
package entity.proj.other
In my Makefile, I tried to generate the right *.pb.go at the right place:
# Proto sources
PROTO= $(wildcard ${PROTODIR}/*.proto)
PBGO= $(PROTO:.proto=.pb.go)
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
#GO_OPT_FLAG= --go_opt=paths=import
#GRPC_OPT_FLAG= --go-grpc_opt=paths=import
.PHONY: clean install proto
## Builds the project
build: proto
go build ${LDFLAGS} -o ${BINARY}
$(PROTOBUF_GO_PLUGIN):
go install google.golang.org/protobuf/cmd/protoc-gen-go#latest
$(GRPC_GO_PLUGIN):
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN)
protoc --proto_path=${PROTODIR} --go_out=. ${GO_OPT_FLAG} --go-grpc_out=. ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
So, depending on the option used with the protoc compiler:
→ With --go_opt=paths=import
A folder tree company.tld/proj/projpb is created by protoc at the project's root. Each object is in a package called projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, have the import path import other "company.tld/proj/projpb/other" (which is brought by the go_package option, but which is wrong because it is not an existing module - go mod tidy/vendor is complaining that it cannot find it).
Normal project files need the following import path to reach the Generated Proto objects :
import pb "company.tld/proj/subproj/company.tld/proj/projpb"
which seems odd and not the proper way to do.
→ With --go_opt=module=company.tld/proj
A folder projpb is created by protoc at the project's root and each generated .pb.go has the package projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, still have the import path import other "company.tld/proj/projpb/other" (which is still brought by the go_package option and is still wrong because this is still a non-existing module - these are generated files... why would I want to create a module of these ?).
The cool thing is that with this go_opt, accessing generated types looks much more normal with
import pb "company.tld/proj/subproj/projpb".
Finally, I tried
using local import path on the go_package option in the .proto files (that is refused on build time, because there would be an import other "./projpb/other" in generated protobuffer object)
to use the replace instruction in the go.mod file like this :
replace (
company.tld/proj/projpb => ./projpb
company.tld/proj/projpb/other => ./projpb/other
)
(but go mod tidy/vendor is complaining that it cannot find the go.mod file inside the generated folder ./projpb)
Has someone encountered a similar problem? Or am I missing a command option to tell to Go, «I generate protobuffer objects in a package, or package in a package, and I simply want to use them. They are not a module, so please, provide the right import paths to the generated object and let me use them in my code».
[Update 01]
I gave a try to the go_opt=paths=source_relative (inspired by this ticket).
I created the folder in the Makefile, protoc generates files inside.
Notes:
generated protos use the full path, specified with the go_package option, to relate to one another.
As long as go_package option needs a full path, Go (go mod tidy/vendor) will want to search for a go.mod file inside the created folder, containing generated protos.
What is the correct way to tell Go that I am not looking for a Module, yet still satisfy the go_package option's full path constraint in the protobuffer file ?
After changing a numerous amount of time the go_package option in the proto files, changing the go_opt on the protoc compiler command, the only way I found to compile my project with my generated protobuffers, respecting every Go constraints, is by creating a go.mod file on-the-fly...
final proto «header» (respects the full puth in the go_package option)
syntax = "proto3";
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
// or for subpackages...
option csharp_namespace = "Proj.Proto.Other";
option go_package = "company.tld/proj/projpb/other";
my Makefile (creates a go.mod file for the generated proto files)
# Proto sources
PROTO= $(shell find ${PROTODIR} -type f -name '*.proto')
PBGO= $(PROTO:.proto=.pb.go)
DEST_DIR=.
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
PROTO_PKG_DIR=projpb
PROTO_MODULE_NAME=${MODULE_NAME}/${PROTO_PKG_DIR}
PROTO_GOMOD_FILE=${PROTO_PKG_DIR}/go.mod
.PHONY: clean install proto gomod
build: proto gomod
go build ${LDFLAGS} -o ${BINARY}
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN) $(DEST_DIR)
${PROTOC} --proto_path=${PROTODIR} --go_out=${DEST_DIR} ${GO_OPT_FLAG} --go-grpc_out=${DEST_DIR} ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
gomod: ${PROTO_GOMOD_FILE}
${PROTO_GOMOD_FILE}:
cd ${PROTO_PKG_DIR} && go mod init ${PROTO_MODULE_NAME} && cd ..
my main go.mod file (redirects the on-the-fly-created module to a local folder inside the project's scope)
module company.tld/proj/subproj
go 1.16
require (
// ...
company.tld/proj/projpb v0.0.0
)
replace company.tld/proj/projpb v0.0.0 => ./projpb
Thanks to the replace instruction, go mod tidy/vendor is happy and do not try to search the module in a remote repository.
Generated *.pb.go files have the right import path : company.tld/proj/projpb (and company.tld/proj/projpb/other for the subpackages).
And the import statement to use generated protos are working fine in the main project.
I hoped there was a simpler and more prettier solution, but alas...
Sorry for the any and thanks to those who gave it a thought !
I'm using Coreos OIDC library and would like to know how to reference (in go.mod file) a branch, since they don't develop under master but use v2 instead.
I tried github.com/coreos/go-oidc#v2 but I get:
go: github.com/coreos/go-oidc#v2#v2.0.0+incompatible: invalid github.com/ import path "github.com/coreos/go-oidc#v2"
go: error loading module requirements
The phrase import path in the error message suggests that somewhere in your code you have written something like:
import "github.com/coreos/go-oidc#v2"
But the import path of a Go package does not include its version: only the entries in the go.mod and go.sum files do.
So instead you should write:
import "github.com/coreos/go-oidc"
and update your go.mod and go.sum files by running
go get -d github.com/coreos/go-oidc#v2
That should result in an entry in your go.mod file like:
require github.com/coreos/go-oidc v2.0.0+incompatible
I'd like to use a helper package from Go Cloud Function. The package has some helper logic that can be shared between multiple functions. But, what is the right way to structure the packages so they all work? The package should be in the same project - not published and public as a completely separate package.
I work at Google. The goal for this question is to proactively answer common questions and help developers starting off with Go on GCF.
You can use subpackages with Go modules. Go modules are Go's new dependency management solution - they let you work outside of GOPATH and let you manage the exact versions of each dependency you have.
Modules also let you define a group of Go packages with the same import path prefix. When you're writing a function, this lets you import other packages in your module.
The function you're deploying needs to be at the root of your module.
Here is an example file structure and how packages would be imported:
.
├── cmd
│ └── main.go # Useful for testing. Can import and setup your function.
├── function.go # Can import example.com/foo/helperpackage
├── function_test.go
├── go.mod # module example.com/foo
└── helperpackage
└── helper.go
This setup has your function in function.go and tested by function_test.go. They are in a module named example.com/foo. helperpackage can be imported by function.go using example.com/foo/helperpackage.
This also has a cmd directory, which may be helpful for local testing. You can import example.com/foo and start an HTTP server which registers your function as an HTTP handler. For example:
package main
import (
"log"
"net/http"
"example.com/foo"
)
func main() {
http.Handle("/HelloHTTP", foo.HelloHTTP)
log.Fatal(http.ListenAndServe(":8080", nil))
}
Note: You could use a vendor directory to achieve the same result. But, all of the packages your function imports would need to be in the vendor directory (with the full import path), which works, but is cumbersome to maintain. It's uncommon to copy sub-packages into your vendor directory, so I wouldn't recommend this.
We also ran into this issue. The go.mod approach provided by #TylerBui-Palsulich (https://stackoverflow.com/a/54255486/2307346) didn't work as we also had to download dependencies from a private repository.
From the Google documentation:
If your function's dependencies are hosted in a repository that is not publicly accessible, you must use a vendor directory to fetch your dependencies before deploying your function
(https://cloud.google.com/functions/docs/writing/specifying-dependencies-go#using_a_vendor_directory)
Given the following package structure:
cloudfunctions.go
sharedpackage
-> shared.go
go.mod
In our go.mod file we defined the following namespace: module some/namespace
The cloudfunctions.go file has the following package definition and imports
package foobar
import (
"bitbucket.org/some/private/dependency"
"some/namespace/sharedpackage"
)
Because of the private dependency we cannot use the go.mod file. Instead we provide a vendor directory.
Warning: If you have both a go.mod file and a vendor directory at the root of your project, the contents of the vendor directory will be ignored when your function is built in the cloud. To ensure that your vendor directory is used, you must exclude the go.mod file from your project's source code prior to deployment. If you are using the gcloud command-line tool, you can ensure that go.mod is not uploaded by using .gcloudignore.
(https://cloud.google.com/functions/docs/writing/specifying-dependencies-go#using_a_vendor_directory)
gcloud deploy will ignore the vendor directory if it finds go.mod, we fix this with a .gloudignore file with the following content:
go.mod
go.sum
At the end we have the following file structure:
cloudfunctions.go
sharedpackage
-> shared.go
go.mod
vendor
.gcloudignore
The last step, Fixing the cannot find package error
When running the deploy step you will still run into an error similar to:
(gcloud.functions.deploy) OperationError: code=3, message=Build failed: /tmp/sgb/gopath/src/serverlessapp/vendor/some/namespace/cloud_functions.go:5:2: cannot find package "some/namespace/sharedpackage" in any of:...
This is because the go.mod file, containing the module name, is ignored. now the Go compiler no longer knows that some/namespace/sharedpackage refers to the local sharedpackage directory.
We managed to get it working by changing the module name to match the package name:
Change module name in go.mod to module foobar
Change the imports in cloudfunctions.go to: "foobar/sharedpackage":
package foobar
import (
"bitbucket.org/some/private/dependency"
"foobar/sharedpackage"
)
Now the Go compiler is able to detect that foobar/sharedpackage is a subpackage of the foobar package.
I have a directory structure like this:
Animal/
dog/
chiwawa.go
rabbit.go
Both chiwawa.go and rabbit.go are packaged named: "package animal" at the top of the file. However when I try to use a variable var Food = apple from chiwawa.go in rabbit.go, I get error - undefined: Food.
I don't have any problems when I place chiwawa.go directly under Animal without the dog directory.
Is this intentional in Go? Can I work around this while retaining the package names and directory structure?
How to Write Go Code
Workspaces
The go tool is designed to work with open source code maintained in
public repositories. Although you don't need to publish your code, the
model for how the environment is set up works the same whether you do
or not.
Go code must be kept inside a workspace. A workspace is a directory
hierarchy with three directories at its root:
src contains Go source files organized into packages (one package per directory),
pkg contains package objects, and
bin contains executable commands.
The go tool builds source packages and installs the resulting binaries
to the pkg and bin directories.
The src subdirectory typically contains multiple version control
repositories (such as for Git or Mercurial) that track the development
of one or more source packages.
To give you an idea of how a workspace looks in practice, here's an
example:
bin/
hello # command executable
outyet # command executable
pkg/
linux_amd64/
github.com/golang/example/
stringutil.a # package object
src/
github.com/golang/example/
.git/ # Git repository metadata
hello/
hello.go # command source
outyet/
main.go # command source
main_test.go # test source
stringutil/
reverse.go # package source
reverse_test.go # test source
This workspace contains one repository (example) comprising two
commands (hello and outyet) and one library (stringutil).
A typical workspace would contain many source repositories containing
many packages and commands. Most Go programmers keep all their Go
source code and dependencies in a single workspace.
For the Go tools, all the Go source files for a Go package should be in the same directory.
It is intentional. Packages in Go are imported using a directory path. So, "animal" and "animal/dog" are two different packages even if you give them the same package name.
To demonstrate this, I wrote the following example set of files:
~/go/src/test/
├── animal
│ ├── animal.go
│ └── dog
│ └── dog.go
└── test.go
animal/animal.go
package animal
type Species string
animal/dog/dog.go
package animal
// In package "animal" importing a package "animal"
import "test/animal"
type Dog struct {
Species animal.Species
Breed string
}
test.go
package main
import (
"fmt"
"test/animal/dog"
)
func main() {
d := animal.Dog{}
fmt.Printf("%#v", d)
}
Notice that dog.go imports the "test/animal" package and references it as "animal".
Furthermore notice that test.go imports "test/animal/dog" to get access to the "Dog" struct type. If it imports, "test/animal", the error (undefined: animal.Dog) is generated.