I'm using protoc to create some DTOs. The definitions are in the following structure:
/protobuf
|-- common.proto
|-- /api
|-- /service
|-- csvdownload.proto
My csvdownload.proto looks like this:
syntax = "proto3";
package protobuf.api.service;
import "common.proto";
option go_package = ".;service"; // golang
message CsvExportRequest {
Common.Currency exportCurrency = 2;
Common.Decimal rounding = 3;
}
and the stub of common.proto looks like this:
syntax = "proto3";
package protobuf;
option go_package = ".;gopb"; // golang
I'm attempt to compile csvdownload.proto by running the following command from the /protobuf directory:
protoc --go_out=gopb --go_opt=paths=source_relative .\api\service\csvdownload.proto
However, I'm getting the following error:
protoc-gen-go: Go package "." has inconsistent names gopb (common.proto) and service (api/service/csvdownload.proto)
I assume this to mean that the code cannot be generated because common.proto and csvdownload.proto have declared different packages but I'm not sure that should make a difference and, from my understanding of how Protobuf works, it shouldn't hinder my ability to compile csvdownload.proto. What am I doing wrong here?
Any help in dealing with this issue would be greatly appreciated.
You must change your directory structure to something like this:
/protobuf
|-- /common
|-- common.proto
|-- /api
|-- /service
|-- csvdownload.proto
also, I suggest you add a real package address for common.proto
syntax = "proto3";
package protobuf;
option go_package = "myProject.com/proto/common"; // golang
then you can import this common to your other protos like this.
import "common/common.proto";
the generating code is(it is the full request you want(I think))
protoc --proto_path=../base_directory/protobuf/api/service/ --proto_path=../base_directory/protobuf/ --go_out=plugins=grpc:./the/generated/path/you/want/directory ../base_directory/protobuf/api/service/csvdownload.proto
Related
I'm trying to make my Go project using hexagonal architecture as described here.
In my project I'm using a gRPC communication generated with protoc from .proto file.
The directories structure:
|- grpc.proto
|-internal
|-core
|-domain
|-services
|- grpcprotocol
And my grpc.proto file has go_package option that points to a specific directory in my Go project
syntax = "proto3";
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
...
Using protoc --go_out=internal/core/domain --go_opt=paths=source_relative --go-grpc_out=internal/core/services/grpcprotocol --go-grpc_opt=paths=source_relative ports.proto I'm able to generate both grpc.pb.go file in internal/core/domain directory and grpc_grpc.pb.go file inside internal/core/services/grpcprotocol directory.
However, grpc.pb.go has a go package named grpcprotocol while it should have a package named domain (I also use other types defined there in separate Go files).
Also grpc_grpc.pb.go file has code that uses types defined in grpc.pb.go without imports (it treats it as if it was defined in the same package).
Is it possible to split those two files into separate Go packages and enforce code that's in grpc_grpc.pb.go to import types from domain directory instead of treating them as defined in the same package?
Your best solution here is too separate the code that you want in grpcprotocol and the one you want into domain, into separate files. Such as:
domain.proto
syntax = "proto3";
package domain;
option go_package = "github.com/myuser/myrepo/internal/core/domain";
//...
grpc.proto
syntax = "proto3";
package grpcprotocol;
option go_package = "github.com/myuser/myrepo/internal/core/services/grpcprotocol";
//...
Then you could import your domain.proto in your grpc.proto, by simply writing import "domain.proto";, provide a --proto_path if domain.proto and grpc.proto are not in the same directory. And finally to reference an object from domain.proto in grpc.proto you can write:
domain.AnObject
After that you can take advantage of the --go_opt=module and --go-grpc_opt=module, to strip the module name in go_package and generate the code at the right place. Like:
protoc --go_out=. --go_opt=module=github.com/myuser/myrepo --go-grpc_out=. --go-grpc_opt=module=github.com/myuser/myrepo *.proto
What this will do is that, it will remove github.com/myuser/myrepo from each go_package and basically start from the root of your module. That's why you can do a --go_out=. and --go-grpc_out=..
Hope that helps, let me know how I can further improve my answer.
Notes
Protobuf package and go_package are not the same. The former is only used for protobuf in order to give context and it extends the qualified name. The go_package is used during go code generation.
The package in the proto file is optional, it makes things more clear and nobody can misuse you proto file without specifying the fully qualified name (little bit safer, if named properly).
I am using gRPC/protobufs as the protocol to communicate between my client and server, both written in go. I'm able to run the command show below to generate the cards.pb.go (server) and cards_grpc.pb.go (client) files without any problem. The server file is working perfectly, without any issues. The client file, however, does not seem to have access to the message items that I have defined within my cards.proto file. My services, as well as my client code, require the defined message struct in order to call the service methods, but I'm not sure what I'm missing.
Here is the command I'm running:
protoc -I="./protos" \
--go_out=plugins=grpc:./server \
--go-grpc_out=./client \
protos/*.proto
Here is my project file structure:
|-- client
|-- protos (generated protobufs for client)
|-- cards_grpc.pb.go (this compiled successfully, but structs representing my messages cannot be found)
|-- protos (This is where the proto files are defined)
|-- cards.proto
|-- server
|-- protos (generated protobufs for server)
|-- cards.pb.go (this is working perfectly, has compiled services and messages)
Note: I have defined option go_package = "./protos"; in my cards.proto file, which is why the generated files have outputted into */protos/*.pb.go locations
So you are not generating any protobuf related code for the client code here, only gRPC one. In order to generate the structure that you are looking for, use the following command:
protoc -I./protos \
--go_out=./server \
--go-grpc_out=./server \
--go_out=./client \
--go-grpc_out=./client \
protos/*.proto
The --go_out generates the go code for protobuf and the --go-grpc_out generates the go code for gRPC.
Another thing, --go_out=plugins=grpc are not supported in go anymore. You should use the --go-grpc_out.
More recommendations
I highly recommend to share the proto directory with both the client and the server (if possible), this limits the potential error due to unsynchronised Proto files.
So you would have something like:
|-- client
|-- protos
|-- cards.proto
|-- cards_grpc.pb.go
|-- cards.pb.go
|-- server
and then both access the files needed.
Second, if you are working with Go modules, I recommend that you use the go_package as following:
option go_package = "${YOUR_MODULE}/protos"
and then generate the code like this:
protoc -Iprotos \
--go_opt=module=${YOUR_MODULE} --go_out=. \
--go-grpc_opt=module=${YOUR_MODULE} --go-grpc_out=. \
protos/*.proto
Notice the . for the --go_out and --go-grpc_out. This maps the root of your project to the module name and this will generate the code inside your protos directory by removing the Module name to the go_package option. Then you will be able to access this generated code in your code like so:
import (
pb "${YOUR_MODULE}/protos"
)
Clarification
Just to be clear about the go_package, you need to understand one thing: the protobuf package and the go_package are not the same thing, the former defines package only usable in .proto files, the latter defines the package ... inside your go files. An example:
For Protobuf package
file1.proto
//...
package protos;
message Test {}
//...
file2.proto
//...
//no package definition
message Test2 {
protos.Test a_test = 1;
}
//...
For go_package
go.mod
module my_module
file1.proto (at location: ${ROOT}/protos)
//...
option go_package = "my_module/protos"
message Test {}
//...
generation
protoc -I./protos \
--go_out=./server \
--go-grpc_out=./server \
--go_out=./client \
--go-grpc_out=./client \
protos/file1.proto
main.go
package main
import (
pb "my_module/proto"
)
func main() {
var test pb.Test;
}
I encounter an issue with Go Module management and a generation of protobuffers (using go1.16, protoc-gen-go#latest).
I have this project structure:
subproj
├── go.mod (module company.tld/proj/subproj)
├── subproj.go (entry point : package main)
├── proto (folder containing .proto files)
├── packageFolder
| └── file1.go (package packageFolder)
└── Makefile (used to generate *.pb.go and build subproj binary)
The proto folder is used by other projects (obviously...) (via git submodule).
Protos are like the following:
syntax = "proto3"
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
package entity.proj
...
because of different version of messages, few protobuffer files need to be in another "namespace":
option go_package = "company.tld/proj/projpb/other";
package entity.proj.other
In my Makefile, I tried to generate the right *.pb.go at the right place:
# Proto sources
PROTO= $(wildcard ${PROTODIR}/*.proto)
PBGO= $(PROTO:.proto=.pb.go)
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
#GO_OPT_FLAG= --go_opt=paths=import
#GRPC_OPT_FLAG= --go-grpc_opt=paths=import
.PHONY: clean install proto
## Builds the project
build: proto
go build ${LDFLAGS} -o ${BINARY}
$(PROTOBUF_GO_PLUGIN):
go install google.golang.org/protobuf/cmd/protoc-gen-go#latest
$(GRPC_GO_PLUGIN):
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc#latest
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN)
protoc --proto_path=${PROTODIR} --go_out=. ${GO_OPT_FLAG} --go-grpc_out=. ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
So, depending on the option used with the protoc compiler:
→ With --go_opt=paths=import
A folder tree company.tld/proj/projpb is created by protoc at the project's root. Each object is in a package called projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, have the import path import other "company.tld/proj/projpb/other" (which is brought by the go_package option, but which is wrong because it is not an existing module - go mod tidy/vendor is complaining that it cannot find it).
Normal project files need the following import path to reach the Generated Proto objects :
import pb "company.tld/proj/subproj/company.tld/proj/projpb"
which seems odd and not the proper way to do.
→ With --go_opt=module=company.tld/proj
A folder projpb is created by protoc at the project's root and each generated .pb.go has the package projpb or other, in the subpackage other.
Generated Proto objects, that include the other namespace-d objects, still have the import path import other "company.tld/proj/projpb/other" (which is still brought by the go_package option and is still wrong because this is still a non-existing module - these are generated files... why would I want to create a module of these ?).
The cool thing is that with this go_opt, accessing generated types looks much more normal with
import pb "company.tld/proj/subproj/projpb".
Finally, I tried
using local import path on the go_package option in the .proto files (that is refused on build time, because there would be an import other "./projpb/other" in generated protobuffer object)
to use the replace instruction in the go.mod file like this :
replace (
company.tld/proj/projpb => ./projpb
company.tld/proj/projpb/other => ./projpb/other
)
(but go mod tidy/vendor is complaining that it cannot find the go.mod file inside the generated folder ./projpb)
Has someone encountered a similar problem? Or am I missing a command option to tell to Go, «I generate protobuffer objects in a package, or package in a package, and I simply want to use them. They are not a module, so please, provide the right import paths to the generated object and let me use them in my code».
[Update 01]
I gave a try to the go_opt=paths=source_relative (inspired by this ticket).
I created the folder in the Makefile, protoc generates files inside.
Notes:
generated protos use the full path, specified with the go_package option, to relate to one another.
As long as go_package option needs a full path, Go (go mod tidy/vendor) will want to search for a go.mod file inside the created folder, containing generated protos.
What is the correct way to tell Go that I am not looking for a Module, yet still satisfy the go_package option's full path constraint in the protobuffer file ?
After changing a numerous amount of time the go_package option in the proto files, changing the go_opt on the protoc compiler command, the only way I found to compile my project with my generated protobuffers, respecting every Go constraints, is by creating a go.mod file on-the-fly...
final proto «header» (respects the full puth in the go_package option)
syntax = "proto3";
option csharp_namespace = "Proj.Proto";
option go_package = "company.tld/proj/projpb";
// or for subpackages...
option csharp_namespace = "Proj.Proto.Other";
option go_package = "company.tld/proj/projpb/other";
my Makefile (creates a go.mod file for the generated proto files)
# Proto sources
PROTO= $(shell find ${PROTODIR} -type f -name '*.proto')
PBGO= $(PROTO:.proto=.pb.go)
DEST_DIR=.
MODULE_NAME=company.tld/proj
GO_OPT_FLAG= --go_opt=module=${MODULE_NAME}
GRPC_OPT_FLAG= --go-grpc_opt=module=${MODULE_NAME}
PROTO_PKG_DIR=projpb
PROTO_MODULE_NAME=${MODULE_NAME}/${PROTO_PKG_DIR}
PROTO_GOMOD_FILE=${PROTO_PKG_DIR}/go.mod
.PHONY: clean install proto gomod
build: proto gomod
go build ${LDFLAGS} -o ${BINARY}
%.pb.go: %.proto | $(PROTOBUF_GO_PLUGIN) $(GRPC_GO_PLUGIN) $(DEST_DIR)
${PROTOC} --proto_path=${PROTODIR} --go_out=${DEST_DIR} ${GO_OPT_FLAG} --go-grpc_out=${DEST_DIR} ${GRPC_OPT_FLAG} $<
proto: $(PBGO)
gomod: ${PROTO_GOMOD_FILE}
${PROTO_GOMOD_FILE}:
cd ${PROTO_PKG_DIR} && go mod init ${PROTO_MODULE_NAME} && cd ..
my main go.mod file (redirects the on-the-fly-created module to a local folder inside the project's scope)
module company.tld/proj/subproj
go 1.16
require (
// ...
company.tld/proj/projpb v0.0.0
)
replace company.tld/proj/projpb v0.0.0 => ./projpb
Thanks to the replace instruction, go mod tidy/vendor is happy and do not try to search the module in a remote repository.
Generated *.pb.go files have the right import path : company.tld/proj/projpb (and company.tld/proj/projpb/other for the subpackages).
And the import statement to use generated protos are working fine in the main project.
I hoped there was a simpler and more prettier solution, but alas...
Sorry for the any and thanks to those who gave it a thought !
I have a library called myProtos which looks like this
.
|-- proto
|---- hello.proto
|
|-- generated
└---- hello.pb.go
I have a .proto file outside called example.proto that should import hello.proto
So the top of the file looks like this:
syntax = "proto3";
package example;
import "path/to/myProtos/proto/hello.proto"
Now, when I compile example.proto I get an import error on example.pb.go because it has the import line import "path/to/myProtos/proto/hello.pb.go"
I tried adding both import paths, but I get an 'import but not used error'. I also tried doing relative imports and passing both directories as flags to protoc, which worked, but I need the import path in the go file to be absolute.
How can I tell protoc that on the go file the path is different?
Is there a better 'best practice' in this case?
What worked for me is to define option go_package = "github.com/<your-account>/<your-cool-project>/<sub-dirs>
Let's assume you have the folder structure your stated:
.
|-- proto
|---- hello.proto
|
|-- generated
└---- hello.pb.go
In your case you would add option go_package = "<path>/<in>/<GOPATH>/generated" to hello.proto.
What is important is that you then have to run
protoc -I. --go_out=$GOPATH ./*.proto
Generally, I would generate the go files alongside the proto files to keep the import paths the same for proto and go files. But this might be a matter of taste. In that case you would then simply set option go_package = "<path>/<in>/<GOPATH>/proto" to hello.proto.
In both cases a relative import of the .proto file should now resolve to the proper import in the generated Go code and the .pb.go files should also be put into the proper Go package folders.
Use package generated; inside your hello.proto file.
Then, protoc -I proto/ proto/*.proto --go_out=generated will generate a hello.pb.go inside generated folder by the package name of generated.
The package inside the proto files tells the protobuf generator which package to use inside the generated file.
Hi all. I'm very new with Go and Gogland. I have a project
I choose "Run kind" as Package - to run not only main file but a project. Why it cannot find main package??
How to import util.myprinter package to main.go to use it??
Please, help me
First, the general structure of your Go workspace seems to be wrong. You need to make it look more like this:
D:
|-- go_projects
| |-- bin
| |-- pkg
| |-- src
| | |-- FirstSteps
| | | |-- main.go
| | | +-- util
| | | +-- myprinter.go
| | |-- SecondProject
| | |-- ThirdProject
...
Second your import statement seems to be empty, I have no idea how GoLand works but if you want to use whatever is in your myprinter.go file, you will need to import the util package, assuming that the myprinter.go file declares its package as util at the top.
// FirstSteps/main.go
package main
import (
"FirstSteps/util"
)
func main() {
util.MyPrinterFunc()
}
And of course to be able to use anything from util there first must be something...
// FirstSteps/util/myprinter.go
package util
func MyPrinterFunc() {
// do stuff...
}
Edit: I'm sorry, I didn't actually answer your question initially. You're getting the error Cannot find package 'main' because of the wrong workspace setup I already mentioned. The Package path tells GoLand where the package you want to run is relative to the $GOPATH/src directory. So after you've setup your wrokspace correctly, you should set the Package path to FirstSteps since that package's absolute path will be $GOPATH/src/FirstSteps. If, later, you want to run the util package you would specify Package path as FirstSteps/util for GoLand to be able to find it.