I have 2 proto compiled under different go packages but when I register them in a a server and run it, I get :
panic: proto: file "common.proto" is already registered
previously from: "github.com/soft/test-platform.go/common"
currently from: "github.com/soft/proto-asterix/asterix"
Here is common.proto in test-platform repository (in /api folder) :
syntax = "proto3";
package soft.testplatform.common; // here I do defint a UNIQUE package name !
option java_multiple_files = true;
option go_package = "github.com/soft/test-platform.go/common"; // Here I do define a unique go package name !
message MyMessage{
string commandId = 1;
}
As you can see, the package definition for go_package and package do not collide with package from github.com/soft/proto-asterix/asterix. Only the .proto filenames are similar (common.proto).
I generate go files with protoc an protoc-gen-go plugin using the following command :
protoc \
--proto_path=../test-platform/api/ \
--go_out=./common --go_opt=paths=source_relative \
../test-platform/api/common.proto
As per documentation here https://developers.google.com/protocol-buffers/docs/reference/go/faq#fix-namespace-conflict the package and filename should be appended to check for a registration conflict but it does not seem to be the case here.
Has anyone encountered such behavior ? Do I miss something obvious to resolve this package name collision ?
Here is what I tried :
Adding/removing package instruction to common.proto file
Change protoc command to use an absolute (and not relative) proto_path
Protoc version : libprotoc 3.15.7
Protoc go plugin version : protoc-gen-go v1.26.0
The accepted answer is not correct anymore. This commit reverted the "bugfix" as it was different from other gRPC implementations.
My only way to fix this was to rename the file/folder.
Thanks to #blackgreen suggestion, indeed, this was a bug fixed by https://go-review.googlesource.com/c/protobuf/+/301953/
While the next release of protoc-gen-go is out, here is a quick fix for your projects :
Use the fixed protoc-gen-go :
go install google.golang.org/protobuf/cmd/protoc-gen-go#febffdd
Change your imports in your go.mod to match
google.golang.org/protobuf v1.26.1-0.20210525005349-febffdd88e85
You should be good to go !
Related
Not so familiar with Golang, it's probably a stupid mistake I made... But still, I can't for the life of me figure it out.
So, I got a proto3 file (let's call it file.proto), whose header is as follows:
syntax = "proto3";
package [package_name];
option go_package = "github.com/[user]/[repository]";
And I use protoc:
protoc --go_out=$GOPATH/src --go-grpc_out=$GOPATH/src file.proto
So far so good, I end up with two generated files (file.pb.go and file_grpc.pb.go) inside /go/src/github.com/[user]/[repository]/, and they are defined inside the package [package_name].
Then, the code I'm trying to build has the following import:
import (
"github.com/[user]/[repository]/[package_name]"
)
And I naively thought it would work. However, it produces the following error when running go mod tidy:
go: downloading github.com/[user]/[repository] v0.0.0-20211105185458-d7aab96b7629
go: finding module for package github.com/[user]/[repository]/[package_name]
example/xxx imports
github.com/[user]/[repository]/[package_name]: module github.com/[user]/[repository]#latest found (v0.0.0-20211105185458-d7aab96b7629), but does not contain package github.com/[user]/[repository]/[package_name]
Any idea what I'm doing wrong here? Go version is go1.19 linux/amd64 within Docker (golang:1.19-alpine).
Note: I also tried to only import github.com/[user]/[repository], same issue obviously.
UPDATE:
OK so what I do is that I get the proto file from the git repository that only contains the proto file:
wget https://raw.githubusercontent.com/[user]/[repository]/file.proto
Then I generate go files from that file with protoc:
protoc --go_out=. --go-grpc_out=. file.proto
Right now, in current directory, it looks like:
- directory
| - process.go
| - file.proto
| - github.com
| - [user]
| - [repository]
| - file.pb.go
| - file_grpc.pb.go
In that same directory, I run:
go mod init xxx
go mod tidy
CGO_ENABLED=0 go build process.go
The import directive in process.go is as follows:
import (
"xxx/github.com/[user]/[repository]"
)
Now it looks like it finds it, but still getting a gRPC error, which is weird because nothing changed. I still have to figure out if it comes from the issue above or not. Thanks!
Your question is really a number of questions in one; I'll try to provide some info that will help. The initial issue you had was because
At least one file with the .go extension must be present in a directory for it to be considered a package.
This makes sense because importing github.com/[user]/[repository] would be fairly pointless if that repository does not contain any .go files (i.e. the go compiler could not really do anything with the files).
Your options are:
Copy the output from protoc directly into your project folder and change the package declarations to match your package. If you do this there is no need for any imports.
Copy (or set go_out argument to protoc) the output from protoc into a subfolder of your project. The import path will then be the value of the module declaration in your go.mod plus the path from the folder that the go.mod is in (this is what you have done).
Store the files in a repo (on github or somewhere else). This does not need to be the same repo as your .proto files if you "want it to be agnostic" (note that 2 & 3 can be combined if the generated files will only be used within one code base or the repo is accessible to all users).
Option 1 is simple but its often beneficial to keep the generated code separate (makes it clear what you should not edit and improves editor autocomplete etc).
Option 2 is OK (especially if protoc writes the files directly and you set go_package appropriately). However issues may arise when the generated files will be used in multiple modules (e.g. as part of your customers code) and your repo is private. They will need to change go_package before running protoc (or search/replace the package declarations) and importing other .proto files may not work well.
Option 3 is probably the best approach in most situations because this works with the go tooling. You can create github.com/[user]/goproto (or similar) and put all of your generated code in there. To use this your customers just need to import github.com/[user]/goproto (no need to run protoc etc).
Go Modules/package intro
The go spec does not detail the format of import paths, leaving it up to the implementation:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
As you are using go modules (pretty much the default now) the implementations rules for resolving package paths (synonym of import path) can be summarised as:
Each package within a module is a collection of source files in the same directory that are compiled together. A package path is the module path joined with the subdirectory containing the package (relative to the module root). For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html".
So if your "module path" (generally the top line in a go.mod) is set to xxx (go mod init xxx) then you would import the package in subfolder github.com/[user]/[repository] with import xxx/github.com/[user]/[repository] (as you have found). If you got rid of the intervening folders and put the files into the [repository] subfolder (directly off your main folder) then it would be import xxx/[repository]
You will note in the examples above that the module names I used are paths to repo (as opposed to the xxx you used in go mod init xxx). This is intentional because it allows the go tooling to find the package when you import it from a different module. For example if you had used go mod init github.com/[user]/[repository] and option go_package = "github.com/[user]/[repository]/myproto";" then the generated files should go into the myproto folder in your project and you import them with import github.com/[user]/[repository]/myproto.
While you do not have to follow this approach I'd highly recommend it (it will save you from a lot of pain!). It can take a while to understand the go way of doing this, but once you do, it works well and makes it very clear where a package is hosted.
I'm trying to make my Go project using hexagonal architecture as described here.
In my project I'm using a gRPC communication generated with protoc from .proto file.
The directories structure:
|- grpc.proto
|-internal
|-core
|-domain
|-services
|- grpcprotocol
And my grpc.proto file has go_package option that points to a specific directory in my Go project
syntax = "proto3";
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
...
Using protoc --go_out=internal/core/domain --go_opt=paths=source_relative --go-grpc_out=internal/core/services/grpcprotocol --go-grpc_opt=paths=source_relative ports.proto I'm able to generate both grpc.pb.go file in internal/core/domain directory and grpc_grpc.pb.go file inside internal/core/services/grpcprotocol directory.
However, grpc.pb.go has a go package named grpcprotocol while it should have a package named domain (I also use other types defined there in separate Go files).
Also grpc_grpc.pb.go file has code that uses types defined in grpc.pb.go without imports (it treats it as if it was defined in the same package).
Is it possible to split those two files into separate Go packages and enforce code that's in grpc_grpc.pb.go to import types from domain directory instead of treating them as defined in the same package?
Your best solution here is too separate the code that you want in grpcprotocol and the one you want into domain, into separate files. Such as:
domain.proto
syntax = "proto3";
package domain;
option go_package = "github.com/myuser/myrepo/internal/core/domain";
//...
grpc.proto
syntax = "proto3";
package grpcprotocol;
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
//...
Then you could import your domain.proto in your grpc.proto, by simply writing import "domain.proto";, provide a --proto_path if domain.proto and grpc.proto are not in the same directory. And finally to reference an object from domain.proto in grpc.proto you can write:
domain.AnObject
After that you can take advantage of the --go_opt=module and --go-grpc_opt=module, to strip the module name in go_package and generate the code at the right place. Like:
protoc --go_out=. --go_opt=module=github.com/myuser/myrepo --go-grpc_out=. --go-grpc_opt=module=github.com/myuser/myrepo *.proto
What this will do is that, it will remove github.com/myuser/myrepo from each go_package and basically start from the root of your module. That's why you can do a --go_out=. and --go-grpc_out=..
Hope that helps, let me know how I can further improve my answer.
Notes
Protobuf package and go_package are not the same. The former is only used for protobuf in order to give context and it extends the qualified name. The go_package is used during go code generation.
The package in the proto file is optional, it makes things more clear and nobody can misuse you proto file without specifying the fully qualified name (little bit safer, if named properly).
I am unable to compile google's fhir proto for go
1. I was able to generate annotations.pb.go with warnings which can be resolved
protoc --proto_path=proto --go_out=. proto/annotations.proto
2020/05/27 12:42:17 WARNING: Missing 'go_package' option in "annotations.proto",
please specify it with the full Go package path as
a future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.
2. Unfortunately, I am unable to resolve fixing issues surrounding file not found.
Example the profile_config.proto which contained the following imports
import "proto/annotations.proto";
import "proto/r4/core/codes.proto";
import "proto/r4/core/datatypes.proto";
Attempting to executed resulted in "not found"
protoc --proto_path=proto --go_out=. proto/profile_config.proto
proto/annotations.proto: File not found.
proto/r4/core/codes.proto: File not found.
proto/r4/core/datatypes.proto: File not found.
Perhaps these proto files can only be used only with java and any other language would require modification on the files.
Regarding 1., you need to edit every .proto you want to compile and add the go_package option. For instance:
option go_package = "github.com/my-org/my-proj/go/gen/fhir/proto"
Regarding 2., you are setting --proto_path=proto, which leads protoc to search for proto/annotations.proto into the following path:
./proto/proto/annotations.proto
If you don't set this option or set it to --proto_path=. you will be able to compile.
I also suggest taking a look at this pull-request.
I am fairly certain this is operator error and I am at the point I am not thinking clearly.
Here is the setup:
$GOPATH/src/github.com/<company>/<service a>/proto/a.proto
$GOPATH/src/github.com/<company>/<service b>/proto/b.proto
etc.
Now in the proto file I am using imports similar to go (perhaps the issue) such that a.proto has:
import "github.com/<company>/<service b>/b.proto"
I have possibly two separate issues.
I cannot get the import to compile properly using go:generate protoc
I cannot get the output a.pb.go file to be placed in the $GOPATH/src/github.com/<company>/<service a>/proto/ path.
I have attempted multiple configurations probably not in the correct combination.
Using option go_package = "github.com/<company>/<service b>/proto" in each .proto file
Multiple variations of go generate;
go:generate protoc --proto_path=.:$GOPATH/src --go_out=$GOPATH/src a.proto
go:generate protoc --proto_path=.:$GOPATH/src --go_out=. a.proto
go:generate protoc --go_out=import_prefix=github.com/<company>/:. api.proto
I clearly have a poor understanding on how protoc looks at import paths and file outputs. Anyone point me in the direction of what I am doing wrong?
Thanks!
Update #1
In a.proto;
option go_package = "github.com/<company>/<service a>/proto";
import "github.com/<company>/<service b>/proto/b.proto";
and the go generate;
//go:generate protoc --proto_path=$GOPATH/src --go_out=$GOPATH/src/github.com/<company>/<service a>/proto a.proto
Which is called from a go file in the proto directory with the a.proto.
I received the error;
a.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path ch encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).
I have confirmed $GOPATH is to the expected location.
Solution
Thanks to Shivam Jindal for pointing me in the correct direction. The import is exactly as described in his solution. The output was a problem of my misusing both --go_out and option go_package. I used the go_package to specify the output location and --go_out to specify the root similar to --proto_path. Now everything works.
option go_package = "github.com/<company>/<service a>/proto";
and
//go:generate protoc --proto_path=$GOPATH/src/ --go_out=$GOPATH/src/ $GOPATH/src/github.com/<company>/<service a>/proto/a.proto
Thanks!
Firstly, option go_package is not meant for other dependency import at all, it's the Go package name where the new proto bindings for Go (a.pb.go file) will be placed.
Now coming to the output file location, I can see you are using go-generate. Firstly it depends from which directory you are invoking that if the path used in --go_out= is relative path. I would say use absolute paths. If you want to put the output file in that location you mentioned, use --go_out=$GOPATH/src/github.com/<company>/<service a>/proto/ in go-generate.
To correctly import the other file b.proto in your a.proto use the fully qualified import path as you have done. Just that use --proto_path $GOPATH/src in go-generate. Also please update the question with the errors you are seeing in case it does not work.
Please see this for more information on import paths.
tl;dr A repo formerly configured to use GOPATH is now configured for Modules. All's good and better. However, protoc correctly (!) generates Golang code for protobufs defined within the repo in a github.com/path/to/repo/protos structure when I'd now prefer these to be generated in my sources, outside of GOPATH. I'm moving them to resolve this. Is there a better solution?
I have a GitHub repo. For the sake of discussion, let's call it github.com/acme/toolbox. In a subdirectory, I have protobuf files that include:
package acme.toolbox.v1;
option go_package = "github.com/acme/toolbox/protos";
When I was GOPATH'ing, all was well and protoc would generate Golang bindings in $GOPATH/src/github.com/acme/toolbox/protos and my code, importing pb "github.com/acme/toolbox/protos", would work.
Moving to Go Modules hasn't been pain-free but, the benefits outweigh the cost and I'm future-proofing myself and the code.
My issue is that I don't see how I can get protoc to generate the Golang bindings into my arbitrarily and outside of GOPATH located clone.
I'm moving the files after they're generated but this feels... inelegant:
cd ${TOOLBOX}
protoc \
--proto_path=./protos \
--go_out=plugins=grpc:/go/src
./protos/*.proto
mv ${GOPATH}/src/github.com/acme/toolbox/protos/*.go ${TOOLBOX}/protos
Is there a better solution?
The main point of the go_package option is to define what the go package name will be. With that said, it can behave differently depending on what you set it too.
If option go_package is defined to be a valid go package name (e.g. protos), protoc will generate the files in the folder defined by --go_out with that package name. If option go_package is instead a path (e.g. github.com/acme/toolbox/protos), protoc will create the folder structure defined relative to --go_out and place the files there with the package name being the same as the last folder name.
Unless I am mistaken in what you are wanting to do, you can change go_package to be:
option go_package = "protos";
and change your protoc invocation to be:
protoc \
--proto_path=./protos \
--go_out=plugins=grpc:${TOOLBOX}/protos
./protos/*.proto
Doing that, the generated files will be placed in ${TOOLBOX}/protos with the go package package protos.